2025-02-14 17:36:05,763 - training_args.py:2100 - _setup_devices - INFO - PyTorch: setting up devices 2025-02-14 17:36:06,316 - configuration_utils.py:731 - _get_config_dict - INFO - loading configuration file ./checkpoints/longvu_llama3_2/config.json 2025-02-14 17:36:06,319 - configuration_utils.py:800 - from_dict - INFO - Model config CambrianConfig { "_name_or_path": "/tmp/iopath_cache/manifold_cache/tree/users/shenx/finetune/09281004-cambrian_llama3_2_t576_ov", "architectures": [ "CambrianLlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "connect_layer": 2, "connector_depth": 3, "connector_only": true, "dino_threshold": 0.83, "drop_threshold": 0.8, "eos_token_id": [ 128001, 128008, 128009 ], "frame_pos": false, "freeze_mm_mlp_adapter": false, "hidden_act": "silu", "hidden_size": 3072, "highres": true, "highres_connect": false, "image_aspect_ratio": "pad", "image_position": 91, "image_token_len": 144, "initializer_range": 0.02, "intermediate_size": 8192, "is_image_newline": true, "is_st_sampler": false, "lowres_token": 8, "max_position_embeddings": 131072, "mlp_bias": false, "mm_patch_merge_type": "flat", "mm_projector_lr": null, "mm_projector_type": "sva", "mm_use_im_patch_token": false, "mm_use_im_start_end": false, "mm_vision_sampler_lr": null, "mm_vision_select_feature": "patch", "mm_vision_select_layer": -2, "mm_vision_tower_aux_list": [ "siglip/CLIP-ViT-SO400M-14-384", "facebook/dinov2-giant-res378" ], "mm_vision_tower_aux_token_len_list": [ 576, 576 ], "mm_vision_tower_lr": null, "model_type": "cambrian_llama", "num_attention_heads": 24, "num_hidden_layers": 28, "num_key_value_heads": 8, "num_of_vision_sampler_layers": 10, "num_query_group": 1, "pretraining_tp": 1, "query_num_list": [ 144 ], "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 32.0, "high_freq_factor": 4.0, "low_freq_factor": 1.0, "original_max_position_embeddings": 8192, "rope_type": "llama3" }, "rope_theta": 500000.0, "spmd_debug": null, "spmd_fsdp_sharding": null, "spmd_mesh": null, "start_of_vision_sampler_layers": 0, "stride_of_vision_sampler_layers": 3, "tie_word_embeddings": false, "tokenizer_model_max_length": 8192, "tokenizer_padding_side": "right", "torch_dtype": "float32", "transformers_version": "4.43.1", "tune_mm_mlp_adapter": false, "unfreeze_mm_vision_tower": false, "use_cache": false, "use_mm_proj": true, "vision_hidden_size": 1024, "vision_tower_aux_token_len_list": [ 576, 576 ], "vocab_size": 128256 } 2025-02-14 17:36:06,320 - modeling_utils.py:3618 - from_pretrained - INFO - loading weights file ./checkpoints/longvu_llama3_2/pytorch_model.bin 2025-02-14 17:36:06,359 - configuration_utils.py:1038 - from_dict - INFO - Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": [ 128001, 128008, 128009 ], "use_cache": false } 2025-02-14 17:36:06,884 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/config.json 2025-02-14 17:36:06,888 - configuration_utils.py:800 - from_dict - INFO - Model config Dinov2Config { "apply_layernorm": true, "architectures": [ "Dinov2Model" ], "attention_probs_dropout_prob": 0.0, "drop_path_rate": 0.0, "hidden_act": "gelu", "hidden_dropout_prob": 0.0, "hidden_size": 1536, "image_size": 518, "initializer_range": 0.02, "layer_norm_eps": 1e-06, "layerscale_value": 1.0, "mlp_ratio": 4, "model_type": "dinov2", "num_attention_heads": 24, "num_channels": 3, "num_hidden_layers": 40, "out_features": [ "stage40" ], "out_indices": [ 40 ], "patch_size": 14, "qkv_bias": true, "reshape_hidden_states": true, "stage_names": [ "stem", "stage1", "stage2", "stage3", "stage4", "stage5", "stage6", "stage7", "stage8", "stage9", "stage10", "stage11", "stage12", "stage13", "stage14", "stage15", "stage16", "stage17", "stage18", "stage19", "stage20", "stage21", "stage22", "stage23", "stage24", "stage25", "stage26", "stage27", "stage28", "stage29", "stage30", "stage31", "stage32", "stage33", "stage34", "stage35", "stage36", "stage37", "stage38", "stage39", "stage40" ], "torch_dtype": "float32", "transformers_version": "4.43.1", "use_swiglu_ffn": true } 2025-02-14 17:36:08,308 - modeling_utils.py:4450 - _load_pretrained_model - INFO - All model checkpoint weights were used when initializing CambrianLlamaForCausalLM. 2025-02-14 17:36:08,308 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of CambrianLlamaForCausalLM were initialized from the model checkpoint at ./checkpoints/longvu_llama3_2. If your task is similar to the task the model of the checkpoint was trained on, you can already use CambrianLlamaForCausalLM for predictions without further training. 2025-02-14 17:36:08,313 - configuration_utils.py:991 - from_pretrained - INFO - loading configuration file ./checkpoints/longvu_llama3_2/generation_config.json 2025-02-14 17:36:08,314 - configuration_utils.py:1038 - from_dict - INFO - Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": [ 128001, 128008, 128009 ], "temperature": 0.6, "top_p": 0.9 } 2025-02-14 17:36:08,559 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file tokenizer.json 2025-02-14 17:36:08,560 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file added_tokens.json 2025-02-14 17:36:08,560 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file special_tokens_map.json 2025-02-14 17:36:08,560 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file tokenizer_config.json 2025-02-14 17:36:08,794 - tokenization_utils_base.py:2533 - _from_pretrained - INFO - Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 2025-02-14 17:36:09,174 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/config.json 2025-02-14 17:36:09,176 - configuration_utils.py:800 - from_dict - INFO - Model config SiglipVisionConfig { "attention_dropout": 0.0, "hidden_act": "gelu_pytorch_tanh", "hidden_size": 1152, "image_size": 384, "intermediate_size": 4304, "layer_norm_eps": 1e-06, "model_type": "siglip_vision_model", "num_attention_heads": 16, "num_channels": 3, "num_hidden_layers": 27, "patch_size": 14, "transformers_version": "4.43.1" } 2025-02-14 17:36:09,177 - modeling_utils.py:3621 - from_pretrained - INFO - loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/model.safetensors 2025-02-14 17:36:09,334 - modeling_utils.py:4440 - _load_pretrained_model - INFO - Some weights of the model checkpoint at google/siglip-so400m-patch14-384 were not used when initializing SiglipVisionModel: ['logit_bias', 'logit_scale', 'text_model.embeddings.position_embedding.weight', 'text_model.embeddings.token_embedding.weight', 'text_model.encoder.layers.0.layer_norm1.bias', 'text_model.encoder.layers.0.layer_norm1.weight', 'text_model.encoder.layers.0.layer_norm2.bias', 'text_model.encoder.layers.0.layer_norm2.weight', 'text_model.encoder.layers.0.mlp.fc1.bias', 'text_model.encoder.layers.0.mlp.fc1.weight', 'text_model.encoder.layers.0.mlp.fc2.bias', 'text_model.encoder.layers.0.mlp.fc2.weight', 'text_model.encoder.layers.0.self_attn.k_proj.bias', 'text_model.encoder.layers.0.self_attn.k_proj.weight', 'text_model.encoder.layers.0.self_attn.out_proj.bias', 'text_model.encoder.layers.0.self_attn.out_proj.weight', 'text_model.encoder.layers.0.self_attn.q_proj.bias', 'text_model.encoder.layers.0.self_attn.q_proj.weight', 'text_model.encoder.layers.0.self_attn.v_proj.bias', 'text_model.encoder.layers.0.self_attn.v_proj.weight', 'text_model.encoder.layers.1.layer_norm1.bias', 'text_model.encoder.layers.1.layer_norm1.weight', 'text_model.encoder.layers.1.layer_norm2.bias', 'text_model.encoder.layers.1.layer_norm2.weight', 'text_model.encoder.layers.1.mlp.fc1.bias', 'text_model.encoder.layers.1.mlp.fc1.weight', 'text_model.encoder.layers.1.mlp.fc2.bias', 'text_model.encoder.layers.1.mlp.fc2.weight', 'text_model.encoder.layers.1.self_attn.k_proj.bias', 'text_model.encoder.layers.1.self_attn.k_proj.weight', 'text_model.encoder.layers.1.self_attn.out_proj.bias', 'text_model.encoder.layers.1.self_attn.out_proj.weight', 'text_model.encoder.layers.1.self_attn.q_proj.bias', 'text_model.encoder.layers.1.self_attn.q_proj.weight', 'text_model.encoder.layers.1.self_attn.v_proj.bias', 'text_model.encoder.layers.1.self_attn.v_proj.weight', 'text_model.encoder.layers.10.layer_norm1.bias', 'text_model.encoder.layers.10.layer_norm1.weight', 'text_model.encoder.layers.10.layer_norm2.bias', 'text_model.encoder.layers.10.layer_norm2.weight', 'text_model.encoder.layers.10.mlp.fc1.bias', 'text_model.encoder.layers.10.mlp.fc1.weight', 'text_model.encoder.layers.10.mlp.fc2.bias', 'text_model.encoder.layers.10.mlp.fc2.weight', 'text_model.encoder.layers.10.self_attn.k_proj.bias', 'text_model.encoder.layers.10.self_attn.k_proj.weight', 'text_model.encoder.layers.10.self_attn.out_proj.bias', 'text_model.encoder.layers.10.self_attn.out_proj.weight', 'text_model.encoder.layers.10.self_attn.q_proj.bias', 'text_model.encoder.layers.10.self_attn.q_proj.weight', 'text_model.encoder.layers.10.self_attn.v_proj.bias', 'text_model.encoder.layers.10.self_attn.v_proj.weight', 'text_model.encoder.layers.11.layer_norm1.bias', 'text_model.encoder.layers.11.layer_norm1.weight', 'text_model.encoder.layers.11.layer_norm2.bias', 'text_model.encoder.layers.11.layer_norm2.weight', 'text_model.encoder.layers.11.mlp.fc1.bias', 'text_model.encoder.layers.11.mlp.fc1.weight', 'text_model.encoder.layers.11.mlp.fc2.bias', 'text_model.encoder.layers.11.mlp.fc2.weight', 'text_model.encoder.layers.11.self_attn.k_proj.bias', 'text_model.encoder.layers.11.self_attn.k_proj.weight', 'text_model.encoder.layers.11.self_attn.out_proj.bias', 'text_model.encoder.layers.11.self_attn.out_proj.weight', 'text_model.encoder.layers.11.self_attn.q_proj.bias', 'text_model.encoder.layers.11.self_attn.q_proj.weight', 'text_model.encoder.layers.11.self_attn.v_proj.bias', 'text_model.encoder.layers.11.self_attn.v_proj.weight', 'text_model.encoder.layers.12.layer_norm1.bias', 'text_model.encoder.layers.12.layer_norm1.weight', 'text_model.encoder.layers.12.layer_norm2.bias', 'text_model.encoder.layers.12.layer_norm2.weight', 'text_model.encoder.layers.12.mlp.fc1.bias', 'text_model.encoder.layers.12.mlp.fc1.weight', 'text_model.encoder.layers.12.mlp.fc2.bias', 'text_model.encoder.layers.12.mlp.fc2.weight', 'text_model.encoder.layers.12.self_attn.k_proj.bias', 'text_model.encoder.layers.12.self_attn.k_proj.weight', 'text_model.encoder.layers.12.self_attn.out_proj.bias', 'text_model.encoder.layers.12.self_attn.out_proj.weight', 'text_model.encoder.layers.12.self_attn.q_proj.bias', 'text_model.encoder.layers.12.self_attn.q_proj.weight', 'text_model.encoder.layers.12.self_attn.v_proj.bias', 'text_model.encoder.layers.12.self_attn.v_proj.weight', 'text_model.encoder.layers.13.layer_norm1.bias', 'text_model.encoder.layers.13.layer_norm1.weight', 'text_model.encoder.layers.13.layer_norm2.bias', 'text_model.encoder.layers.13.layer_norm2.weight', 'text_model.encoder.layers.13.mlp.fc1.bias', 'text_model.encoder.layers.13.mlp.fc1.weight', 'text_model.encoder.layers.13.mlp.fc2.bias', 'text_model.encoder.layers.13.mlp.fc2.weight', 'text_model.encoder.layers.13.self_attn.k_proj.bias', 'text_model.encoder.layers.13.self_attn.k_proj.weight', 'text_model.encoder.layers.13.self_attn.out_proj.bias', 'text_model.encoder.layers.13.self_attn.out_proj.weight', 'text_model.encoder.layers.13.self_attn.q_proj.bias', 'text_model.encoder.layers.13.self_attn.q_proj.weight', 'text_model.encoder.layers.13.self_attn.v_proj.bias', 'text_model.encoder.layers.13.self_attn.v_proj.weight', 'text_model.encoder.layers.14.layer_norm1.bias', 'text_model.encoder.layers.14.layer_norm1.weight', 'text_model.encoder.layers.14.layer_norm2.bias', 'text_model.encoder.layers.14.layer_norm2.weight', 'text_model.encoder.layers.14.mlp.fc1.bias', 'text_model.encoder.layers.14.mlp.fc1.weight', 'text_model.encoder.layers.14.mlp.fc2.bias', 'text_model.encoder.layers.14.mlp.fc2.weight', 'text_model.encoder.layers.14.self_attn.k_proj.bias', 'text_model.encoder.layers.14.self_attn.k_proj.weight', 'text_model.encoder.layers.14.self_attn.out_proj.bias', 'text_model.encoder.layers.14.self_attn.out_proj.weight', 'text_model.encoder.layers.14.self_attn.q_proj.bias', 'text_model.encoder.layers.14.self_attn.q_proj.weight', 'text_model.encoder.layers.14.self_attn.v_proj.bias', 'text_model.encoder.layers.14.self_attn.v_proj.weight', 'text_model.encoder.layers.15.layer_norm1.bias', 'text_model.encoder.layers.15.layer_norm1.weight', 'text_model.encoder.layers.15.layer_norm2.bias', 'text_model.encoder.layers.15.layer_norm2.weight', 'text_model.encoder.layers.15.mlp.fc1.bias', 'text_model.encoder.layers.15.mlp.fc1.weight', 'text_model.encoder.layers.15.mlp.fc2.bias', 'text_model.encoder.layers.15.mlp.fc2.weight', 'text_model.encoder.layers.15.self_attn.k_proj.bias', 'text_model.encoder.layers.15.self_attn.k_proj.weight', 'text_model.encoder.layers.15.self_attn.out_proj.bias', 'text_model.encoder.layers.15.self_attn.out_proj.weight', 'text_model.encoder.layers.15.self_attn.q_proj.bias', 'text_model.encoder.layers.15.self_attn.q_proj.weight', 'text_model.encoder.layers.15.self_attn.v_proj.bias', 'text_model.encoder.layers.15.self_attn.v_proj.weight', 'text_model.encoder.layers.16.layer_norm1.bias', 'text_model.encoder.layers.16.layer_norm1.weight', 'text_model.encoder.layers.16.layer_norm2.bias', 'text_model.encoder.layers.16.layer_norm2.weight', 'text_model.encoder.layers.16.mlp.fc1.bias', 'text_model.encoder.layers.16.mlp.fc1.weight', 'text_model.encoder.layers.16.mlp.fc2.bias', 'text_model.encoder.layers.16.mlp.fc2.weight', 'text_model.encoder.layers.16.self_attn.k_proj.bias', 'text_model.encoder.layers.16.self_attn.k_proj.weight', 'text_model.encoder.layers.16.self_attn.out_proj.bias', 'text_model.encoder.layers.16.self_attn.out_proj.weight', 'text_model.encoder.layers.16.self_attn.q_proj.bias', 'text_model.encoder.layers.16.self_attn.q_proj.weight', 'text_model.encoder.layers.16.self_attn.v_proj.bias', 'text_model.encoder.layers.16.self_attn.v_proj.weight', 'text_model.encoder.layers.17.layer_norm1.bias', 'text_model.encoder.layers.17.layer_norm1.weight', 'text_model.encoder.layers.17.layer_norm2.bias', 'text_model.encoder.layers.17.layer_norm2.weight', 'text_model.encoder.layers.17.mlp.fc1.bias', 'text_model.encoder.layers.17.mlp.fc1.weight', 'text_model.encoder.layers.17.mlp.fc2.bias', 'text_model.encoder.layers.17.mlp.fc2.weight', 'text_model.encoder.layers.17.self_attn.k_proj.bias', 'text_model.encoder.layers.17.self_attn.k_proj.weight', 'text_model.encoder.layers.17.self_attn.out_proj.bias', 'text_model.encoder.layers.17.self_attn.out_proj.weight', 'text_model.encoder.layers.17.self_attn.q_proj.bias', 'text_model.encoder.layers.17.self_attn.q_proj.weight', 'text_model.encoder.layers.17.self_attn.v_proj.bias', 'text_model.encoder.layers.17.self_attn.v_proj.weight', 'text_model.encoder.layers.18.layer_norm1.bias', 'text_model.encoder.layers.18.layer_norm1.weight', 'text_model.encoder.layers.18.layer_norm2.bias', 'text_model.encoder.layers.18.layer_norm2.weight', 'text_model.encoder.layers.18.mlp.fc1.bias', 'text_model.encoder.layers.18.mlp.fc1.weight', 'text_model.encoder.layers.18.mlp.fc2.bias', 'text_model.encoder.layers.18.mlp.fc2.weight', 'text_model.encoder.layers.18.self_attn.k_proj.bias', 'text_model.encoder.layers.18.self_attn.k_proj.weight', 'text_model.encoder.layers.18.self_attn.out_proj.bias', 'text_model.encoder.layers.18.self_attn.out_proj.weight', 'text_model.encoder.layers.18.self_attn.q_proj.bias', 'text_model.encoder.layers.18.self_attn.q_proj.weight', 'text_model.encoder.layers.18.self_attn.v_proj.bias', 'text_model.encoder.layers.18.self_attn.v_proj.weight', 'text_model.encoder.layers.19.layer_norm1.bias', 'text_model.encoder.layers.19.layer_norm1.weight', 'text_model.encoder.layers.19.layer_norm2.bias', 'text_model.encoder.layers.19.layer_norm2.weight', 'text_model.encoder.layers.19.mlp.fc1.bias', 'text_model.encoder.layers.19.mlp.fc1.weight', 'text_model.encoder.layers.19.mlp.fc2.bias', 'text_model.encoder.layers.19.mlp.fc2.weight', 'text_model.encoder.layers.19.self_attn.k_proj.bias', 'text_model.encoder.layers.19.self_attn.k_proj.weight', 'text_model.encoder.layers.19.self_attn.out_proj.bias', 'text_model.encoder.layers.19.self_attn.out_proj.weight', 'text_model.encoder.layers.19.self_attn.q_proj.bias', 'text_model.encoder.layers.19.self_attn.q_proj.weight', 'text_model.encoder.layers.19.self_attn.v_proj.bias', 'text_model.encoder.layers.19.self_attn.v_proj.weight', 'text_model.encoder.layers.2.layer_norm1.bias', 'text_model.encoder.layers.2.layer_norm1.weight', 'text_model.encoder.layers.2.layer_norm2.bias', 'text_model.encoder.layers.2.layer_norm2.weight', 'text_model.encoder.layers.2.mlp.fc1.bias', 'text_model.encoder.layers.2.mlp.fc1.weight', 'text_model.encoder.layers.2.mlp.fc2.bias', 'text_model.encoder.layers.2.mlp.fc2.weight', 'text_model.encoder.layers.2.self_attn.k_proj.bias', 'text_model.encoder.layers.2.self_attn.k_proj.weight', 'text_model.encoder.layers.2.self_attn.out_proj.bias', 'text_model.encoder.layers.2.self_attn.out_proj.weight', 'text_model.encoder.layers.2.self_attn.q_proj.bias', 'text_model.encoder.layers.2.self_attn.q_proj.weight', 'text_model.encoder.layers.2.self_attn.v_proj.bias', 'text_model.encoder.layers.2.self_attn.v_proj.weight', 'text_model.encoder.layers.20.layer_norm1.bias', 'text_model.encoder.layers.20.layer_norm1.weight', 'text_model.encoder.layers.20.layer_norm2.bias', 'text_model.encoder.layers.20.layer_norm2.weight', 'text_model.encoder.layers.20.mlp.fc1.bias', 'text_model.encoder.layers.20.mlp.fc1.weight', 'text_model.encoder.layers.20.mlp.fc2.bias', 'text_model.encoder.layers.20.mlp.fc2.weight', 'text_model.encoder.layers.20.self_attn.k_proj.bias', 'text_model.encoder.layers.20.self_attn.k_proj.weight', 'text_model.encoder.layers.20.self_attn.out_proj.bias', 'text_model.encoder.layers.20.self_attn.out_proj.weight', 'text_model.encoder.layers.20.self_attn.q_proj.bias', 'text_model.encoder.layers.20.self_attn.q_proj.weight', 'text_model.encoder.layers.20.self_attn.v_proj.bias', 'text_model.encoder.layers.20.self_attn.v_proj.weight', 'text_model.encoder.layers.21.layer_norm1.bias', 'text_model.encoder.layers.21.layer_norm1.weight', 'text_model.encoder.layers.21.layer_norm2.bias', 'text_model.encoder.layers.21.layer_norm2.weight', 'text_model.encoder.layers.21.mlp.fc1.bias', 'text_model.encoder.layers.21.mlp.fc1.weight', 'text_model.encoder.layers.21.mlp.fc2.bias', 'text_model.encoder.layers.21.mlp.fc2.weight', 'text_model.encoder.layers.21.self_attn.k_proj.bias', 'text_model.encoder.layers.21.self_attn.k_proj.weight', 'text_model.encoder.layers.21.self_attn.out_proj.bias', 'text_model.encoder.layers.21.self_attn.out_proj.weight', 'text_model.encoder.layers.21.self_attn.q_proj.bias', 'text_model.encoder.layers.21.self_attn.q_proj.weight', 'text_model.encoder.layers.21.self_attn.v_proj.bias', 'text_model.encoder.layers.21.self_attn.v_proj.weight', 'text_model.encoder.layers.22.layer_norm1.bias', 'text_model.encoder.layers.22.layer_norm1.weight', 'text_model.encoder.layers.22.layer_norm2.bias', 'text_model.encoder.layers.22.layer_norm2.weight', 'text_model.encoder.layers.22.mlp.fc1.bias', 'text_model.encoder.layers.22.mlp.fc1.weight', 'text_model.encoder.layers.22.mlp.fc2.bias', 'text_model.encoder.layers.22.mlp.fc2.weight', 'text_model.encoder.layers.22.self_attn.k_proj.bias', 'text_model.encoder.layers.22.self_attn.k_proj.weight', 'text_model.encoder.layers.22.self_attn.out_proj.bias', 'text_model.encoder.layers.22.self_attn.out_proj.weight', 'text_model.encoder.layers.22.self_attn.q_proj.bias', 'text_model.encoder.layers.22.self_attn.q_proj.weight', 'text_model.encoder.layers.22.self_attn.v_proj.bias', 'text_model.encoder.layers.22.self_attn.v_proj.weight', 'text_model.encoder.layers.23.layer_norm1.bias', 'text_model.encoder.layers.23.layer_norm1.weight', 'text_model.encoder.layers.23.layer_norm2.bias', 'text_model.encoder.layers.23.layer_norm2.weight', 'text_model.encoder.layers.23.mlp.fc1.bias', 'text_model.encoder.layers.23.mlp.fc1.weight', 'text_model.encoder.layers.23.mlp.fc2.bias', 'text_model.encoder.layers.23.mlp.fc2.weight', 'text_model.encoder.layers.23.self_attn.k_proj.bias', 'text_model.encoder.layers.23.self_attn.k_proj.weight', 'text_model.encoder.layers.23.self_attn.out_proj.bias', 'text_model.encoder.layers.23.self_attn.out_proj.weight', 'text_model.encoder.layers.23.self_attn.q_proj.bias', 'text_model.encoder.layers.23.self_attn.q_proj.weight', 'text_model.encoder.layers.23.self_attn.v_proj.bias', 'text_model.encoder.layers.23.self_attn.v_proj.weight', 'text_model.encoder.layers.24.layer_norm1.bias', 'text_model.encoder.layers.24.layer_norm1.weight', 'text_model.encoder.layers.24.layer_norm2.bias', 'text_model.encoder.layers.24.layer_norm2.weight', 'text_model.encoder.layers.24.mlp.fc1.bias', 'text_model.encoder.layers.24.mlp.fc1.weight', 'text_model.encoder.layers.24.mlp.fc2.bias', 'text_model.encoder.layers.24.mlp.fc2.weight', 'text_model.encoder.layers.24.self_attn.k_proj.bias', 'text_model.encoder.layers.24.self_attn.k_proj.weight', 'text_model.encoder.layers.24.self_attn.out_proj.bias', 'text_model.encoder.layers.24.self_attn.out_proj.weight', 'text_model.encoder.layers.24.self_attn.q_proj.bias', 'text_model.encoder.layers.24.self_attn.q_proj.weight', 'text_model.encoder.layers.24.self_attn.v_proj.bias', 'text_model.encoder.layers.24.self_attn.v_proj.weight', 'text_model.encoder.layers.25.layer_norm1.bias', 'text_model.encoder.layers.25.layer_norm1.weight', 'text_model.encoder.layers.25.layer_norm2.bias', 'text_model.encoder.layers.25.layer_norm2.weight', 'text_model.encoder.layers.25.mlp.fc1.bias', 'text_model.encoder.layers.25.mlp.fc1.weight', 'text_model.encoder.layers.25.mlp.fc2.bias', 'text_model.encoder.layers.25.mlp.fc2.weight', 'text_model.encoder.layers.25.self_attn.k_proj.bias', 'text_model.encoder.layers.25.self_attn.k_proj.weight', 'text_model.encoder.layers.25.self_attn.out_proj.bias', 'text_model.encoder.layers.25.self_attn.out_proj.weight', 'text_model.encoder.layers.25.self_attn.q_proj.bias', 'text_model.encoder.layers.25.self_attn.q_proj.weight', 'text_model.encoder.layers.25.self_attn.v_proj.bias', 'text_model.encoder.layers.25.self_attn.v_proj.weight', 'text_model.encoder.layers.26.layer_norm1.bias', 'text_model.encoder.layers.26.layer_norm1.weight', 'text_model.encoder.layers.26.layer_norm2.bias', 'text_model.encoder.layers.26.layer_norm2.weight', 'text_model.encoder.layers.26.mlp.fc1.bias', 'text_model.encoder.layers.26.mlp.fc1.weight', 'text_model.encoder.layers.26.mlp.fc2.bias', 'text_model.encoder.layers.26.mlp.fc2.weight', 'text_model.encoder.layers.26.self_attn.k_proj.bias', 'text_model.encoder.layers.26.self_attn.k_proj.weight', 'text_model.encoder.layers.26.self_attn.out_proj.bias', 'text_model.encoder.layers.26.self_attn.out_proj.weight', 'text_model.encoder.layers.26.self_attn.q_proj.bias', 'text_model.encoder.layers.26.self_attn.q_proj.weight', 'text_model.encoder.layers.26.self_attn.v_proj.bias', 'text_model.encoder.layers.26.self_attn.v_proj.weight', 'text_model.encoder.layers.3.layer_norm1.bias', 'text_model.encoder.layers.3.layer_norm1.weight', 'text_model.encoder.layers.3.layer_norm2.bias', 'text_model.encoder.layers.3.layer_norm2.weight', 'text_model.encoder.layers.3.mlp.fc1.bias', 'text_model.encoder.layers.3.mlp.fc1.weight', 'text_model.encoder.layers.3.mlp.fc2.bias', 'text_model.encoder.layers.3.mlp.fc2.weight', 'text_model.encoder.layers.3.self_attn.k_proj.bias', 'text_model.encoder.layers.3.self_attn.k_proj.weight', 'text_model.encoder.layers.3.self_attn.out_proj.bias', 'text_model.encoder.layers.3.self_attn.out_proj.weight', 'text_model.encoder.layers.3.self_attn.q_proj.bias', 'text_model.encoder.layers.3.self_attn.q_proj.weight', 'text_model.encoder.layers.3.self_attn.v_proj.bias', 'text_model.encoder.layers.3.self_attn.v_proj.weight', 'text_model.encoder.layers.4.layer_norm1.bias', 'text_model.encoder.layers.4.layer_norm1.weight', 'text_model.encoder.layers.4.layer_norm2.bias', 'text_model.encoder.layers.4.layer_norm2.weight', 'text_model.encoder.layers.4.mlp.fc1.bias', 'text_model.encoder.layers.4.mlp.fc1.weight', 'text_model.encoder.layers.4.mlp.fc2.bias', 'text_model.encoder.layers.4.mlp.fc2.weight', 'text_model.encoder.layers.4.self_attn.k_proj.bias', 'text_model.encoder.layers.4.self_attn.k_proj.weight', 'text_model.encoder.layers.4.self_attn.out_proj.bias', 'text_model.encoder.layers.4.self_attn.out_proj.weight', 'text_model.encoder.layers.4.self_attn.q_proj.bias', 'text_model.encoder.layers.4.self_attn.q_proj.weight', 'text_model.encoder.layers.4.self_attn.v_proj.bias', 'text_model.encoder.layers.4.self_attn.v_proj.weight', 'text_model.encoder.layers.5.layer_norm1.bias', 'text_model.encoder.layers.5.layer_norm1.weight', 'text_model.encoder.layers.5.layer_norm2.bias', 'text_model.encoder.layers.5.layer_norm2.weight', 'text_model.encoder.layers.5.mlp.fc1.bias', 'text_model.encoder.layers.5.mlp.fc1.weight', 'text_model.encoder.layers.5.mlp.fc2.bias', 'text_model.encoder.layers.5.mlp.fc2.weight', 'text_model.encoder.layers.5.self_attn.k_proj.bias', 'text_model.encoder.layers.5.self_attn.k_proj.weight', 'text_model.encoder.layers.5.self_attn.out_proj.bias', 'text_model.encoder.layers.5.self_attn.out_proj.weight', 'text_model.encoder.layers.5.self_attn.q_proj.bias', 'text_model.encoder.layers.5.self_attn.q_proj.weight', 'text_model.encoder.layers.5.self_attn.v_proj.bias', 'text_model.encoder.layers.5.self_attn.v_proj.weight', 'text_model.encoder.layers.6.layer_norm1.bias', 'text_model.encoder.layers.6.layer_norm1.weight', 'text_model.encoder.layers.6.layer_norm2.bias', 'text_model.encoder.layers.6.layer_norm2.weight', 'text_model.encoder.layers.6.mlp.fc1.bias', 'text_model.encoder.layers.6.mlp.fc1.weight', 'text_model.encoder.layers.6.mlp.fc2.bias', 'text_model.encoder.layers.6.mlp.fc2.weight', 'text_model.encoder.layers.6.self_attn.k_proj.bias', 'text_model.encoder.layers.6.self_attn.k_proj.weight', 'text_model.encoder.layers.6.self_attn.out_proj.bias', 'text_model.encoder.layers.6.self_attn.out_proj.weight', 'text_model.encoder.layers.6.self_attn.q_proj.bias', 'text_model.encoder.layers.6.self_attn.q_proj.weight', 'text_model.encoder.layers.6.self_attn.v_proj.bias', 'text_model.encoder.layers.6.self_attn.v_proj.weight', 'text_model.encoder.layers.7.layer_norm1.bias', 'text_model.encoder.layers.7.layer_norm1.weight', 'text_model.encoder.layers.7.layer_norm2.bias', 'text_model.encoder.layers.7.layer_norm2.weight', 'text_model.encoder.layers.7.mlp.fc1.bias', 'text_model.encoder.layers.7.mlp.fc1.weight', 'text_model.encoder.layers.7.mlp.fc2.bias', 'text_model.encoder.layers.7.mlp.fc2.weight', 'text_model.encoder.layers.7.self_attn.k_proj.bias', 'text_model.encoder.layers.7.self_attn.k_proj.weight', 'text_model.encoder.layers.7.self_attn.out_proj.bias', 'text_model.encoder.layers.7.self_attn.out_proj.weight', 'text_model.encoder.layers.7.self_attn.q_proj.bias', 'text_model.encoder.layers.7.self_attn.q_proj.weight', 'text_model.encoder.layers.7.self_attn.v_proj.bias', 'text_model.encoder.layers.7.self_attn.v_proj.weight', 'text_model.encoder.layers.8.layer_norm1.bias', 'text_model.encoder.layers.8.layer_norm1.weight', 'text_model.encoder.layers.8.layer_norm2.bias', 'text_model.encoder.layers.8.layer_norm2.weight', 'text_model.encoder.layers.8.mlp.fc1.bias', 'text_model.encoder.layers.8.mlp.fc1.weight', 'text_model.encoder.layers.8.mlp.fc2.bias', 'text_model.encoder.layers.8.mlp.fc2.weight', 'text_model.encoder.layers.8.self_attn.k_proj.bias', 'text_model.encoder.layers.8.self_attn.k_proj.weight', 'text_model.encoder.layers.8.self_attn.out_proj.bias', 'text_model.encoder.layers.8.self_attn.out_proj.weight', 'text_model.encoder.layers.8.self_attn.q_proj.bias', 'text_model.encoder.layers.8.self_attn.q_proj.weight', 'text_model.encoder.layers.8.self_attn.v_proj.bias', 'text_model.encoder.layers.8.self_attn.v_proj.weight', 'text_model.encoder.layers.9.layer_norm1.bias', 'text_model.encoder.layers.9.layer_norm1.weight', 'text_model.encoder.layers.9.layer_norm2.bias', 'text_model.encoder.layers.9.layer_norm2.weight', 'text_model.encoder.layers.9.mlp.fc1.bias', 'text_model.encoder.layers.9.mlp.fc1.weight', 'text_model.encoder.layers.9.mlp.fc2.bias', 'text_model.encoder.layers.9.mlp.fc2.weight', 'text_model.encoder.layers.9.self_attn.k_proj.bias', 'text_model.encoder.layers.9.self_attn.k_proj.weight', 'text_model.encoder.layers.9.self_attn.out_proj.bias', 'text_model.encoder.layers.9.self_attn.out_proj.weight', 'text_model.encoder.layers.9.self_attn.q_proj.bias', 'text_model.encoder.layers.9.self_attn.q_proj.weight', 'text_model.encoder.layers.9.self_attn.v_proj.bias', 'text_model.encoder.layers.9.self_attn.v_proj.weight', 'text_model.final_layer_norm.bias', 'text_model.final_layer_norm.weight', 'text_model.head.bias', 'text_model.head.weight'] - This IS expected if you are initializing SiglipVisionModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing SiglipVisionModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). 2025-02-14 17:36:09,336 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of SiglipVisionModel were initialized from the model checkpoint at google/siglip-so400m-patch14-384. If your task is similar to the task the model of the checkpoint was trained on, you can already use SiglipVisionModel for predictions without further training. 2025-02-14 17:36:09,828 - image_processing_base.py:375 - get_image_processor_dict - INFO - loading configuration file preprocessor_config.json from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/preprocessor_config.json 2025-02-14 17:36:09,829 - image_processing_base.py:429 - from_dict - INFO - Image processor SiglipImageProcessor { "do_convert_rgb": null, "do_normalize": true, "do_rescale": true, "do_resize": true, "image_mean": [ 0.5, 0.5, 0.5 ], "image_processor_type": "SiglipImageProcessor", "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "SiglipProcessor", "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "height": 384, "width": 384 } } 2025-02-14 17:36:10,497 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/config.json 2025-02-14 17:36:10,500 - configuration_utils.py:800 - from_dict - INFO - Model config Dinov2Config { "apply_layernorm": true, "architectures": [ "Dinov2Model" ], "attention_probs_dropout_prob": 0.0, "drop_path_rate": 0.0, "hidden_act": "gelu", "hidden_dropout_prob": 0.0, "hidden_size": 1536, "image_size": 518, "initializer_range": 0.02, "layer_norm_eps": 1e-06, "layerscale_value": 1.0, "mlp_ratio": 4, "model_type": "dinov2", "num_attention_heads": 24, "num_channels": 3, "num_hidden_layers": 40, "out_features": [ "stage40" ], "out_indices": [ 40 ], "patch_size": 14, "qkv_bias": true, "reshape_hidden_states": true, "stage_names": [ "stem", "stage1", "stage2", "stage3", "stage4", "stage5", "stage6", "stage7", "stage8", "stage9", "stage10", "stage11", "stage12", "stage13", "stage14", "stage15", "stage16", "stage17", "stage18", "stage19", "stage20", "stage21", "stage22", "stage23", "stage24", "stage25", "stage26", "stage27", "stage28", "stage29", "stage30", "stage31", "stage32", "stage33", "stage34", "stage35", "stage36", "stage37", "stage38", "stage39", "stage40" ], "torch_dtype": "float32", "transformers_version": "4.43.1", "use_swiglu_ffn": true } 2025-02-14 17:36:10,501 - modeling_utils.py:3621 - from_pretrained - INFO - loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/model.safetensors 2025-02-14 17:36:10,839 - modeling_utils.py:4450 - _load_pretrained_model - INFO - All model checkpoint weights were used when initializing Dinov2Model. 2025-02-14 17:36:10,839 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of Dinov2Model were initialized from the model checkpoint at facebook/dinov2-giant. If your task is similar to the task the model of the checkpoint was trained on, you can already use Dinov2Model for predictions without further training. 2025-02-14 17:36:11,024 - image_processing_base.py:375 - get_image_processor_dict - INFO - loading configuration file preprocessor_config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/preprocessor_config.json 2025-02-14 17:36:11,027 - image_processing_base.py:429 - from_dict - INFO - Image processor BitImageProcessor { "crop_size": { "height": 378, "width": 378 }, "do_center_crop": true, "do_convert_rgb": true, "do_normalize": true, "do_rescale": true, "do_resize": true, "image_mean": [ 0.485, 0.456, 0.406 ], "image_processor_type": "BitImageProcessor", "image_std": [ 0.229, 0.224, 0.225 ], "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "shortest_edge": 378 } } 2025-02-14 17:36:11,804 - finetune_llama.py:1239 - train - INFO - Total params: 3264865280 2025-02-14 17:36:11,804 - finetune_llama.py:1240 - train - INFO - Trainable params: 12589056 2025-02-14 17:36:11,804 - finetune_llama.py:1241 - train - INFO - LM head params: 394002432 2025-02-14 17:36:14,980 - trainer_callback.py:423 - add_callback - WARNING - You are adding a to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is :DefaultFlowCallback TensorBoardCallback 2025-02-14 17:36:14,980 - trainer.py:648 - __init__ - INFO - Using auto half precision backend 2025-02-14 17:36:15,469 - trainer.py:2134 - _inner_training_loop - INFO - ***** Running training ***** 2025-02-14 17:36:15,469 - trainer.py:2135 - _inner_training_loop - INFO - Num examples = 550 2025-02-14 17:36:15,469 - trainer.py:2136 - _inner_training_loop - INFO - Num Epochs = 2 2025-02-14 17:36:15,469 - trainer.py:2137 - _inner_training_loop - INFO - Instantaneous batch size per device = 1 2025-02-14 17:36:15,469 - trainer.py:2140 - _inner_training_loop - INFO - Total train batch size (w. parallel, distributed & accumulation) = 1 2025-02-14 17:36:15,469 - trainer.py:2141 - _inner_training_loop - INFO - Gradient Accumulation steps = 1 2025-02-14 17:36:15,469 - trainer.py:2142 - _inner_training_loop - INFO - Total optimization steps = 1,100 2025-02-14 17:36:15,471 - trainer.py:2143 - _inner_training_loop - INFO - Number of trainable parameters = 406,591,488 2025-02-14 17:39:12,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:39:12,833 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:39:12,862 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:39:12,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:39:12,866 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2879, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:39:12,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:39:12,867 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2879, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:39:57,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:39:57,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:39:57,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.20 seconds 2025-02-14 17:39:57,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:57,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31337.24 MB 2025-02-14 17:39:57,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41559.35 MB 2025-02-14 17:39:57,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10222.11 MB 2025-02-14 17:39:57,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32541.51 MB 2025-02-14 17:39:57,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42427.48 MB 2025-02-14 17:39:57,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9885.97 MB 2025-02-14 17:39:57,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51747.97 MB 2025-02-14 17:39:57,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:39:57,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:39:57,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:39:57,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:57,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41559.35 MB 2025-02-14 17:39:57,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29086.68 MB 2025-02-14 17:39:57,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12472.67 MB 2025-02-14 17:39:57,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42427.48 MB 2025-02-14 17:39:57,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 80182.51 MB 2025-02-14 17:39:57,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 37755.03 MB 2025-02-14 17:39:57,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66558.68 MB 2025-02-14 17:39:59,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:39:59,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:39:59,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 17:39:59,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29086.68 MB 2025-02-14 17:39:59,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29617.52 MB 2025-02-14 17:39:59,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:39:59,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 80182.51 MB 2025-02-14 17:39:59,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34024.19 MB 2025-02-14 17:39:59,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -46158.32 MB 2025-02-14 17:39:59,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33596.07 MB 2025-02-14 17:39:59,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:39:59,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:39:59,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:39:59,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29617.52 MB 2025-02-14 17:39:59,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31506.73 MB 2025-02-14 17:39:59,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-14 17:39:59,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34024.19 MB 2025-02-14 17:39:59,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34970.01 MB 2025-02-14 17:39:59,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 945.82 MB 2025-02-14 17:39:59,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32924.16 MB 2025-02-14 17:39:59,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:39:59,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:39:59,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:39:59,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31506.73 MB 2025-02-14 17:39:59,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33748.58 MB 2025-02-14 17:39:59,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:39:59,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34970.01 MB 2025-02-14 17:39:59,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40634.42 MB 2025-02-14 17:39:59,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 17:39:59,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39292.86 MB 2025-02-14 17:39:59,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:39:59,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:39:59,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:39:59,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29617.52 MB 2025-02-14 17:39:59,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33748.58 MB 2025-02-14 17:39:59,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-14 17:39:59,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34024.19 MB 2025-02-14 17:39:59,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40634.42 MB 2025-02-14 17:39:59,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6610.22 MB 2025-02-14 17:39:59,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39292.86 MB 2025-02-14 17:39:59,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:39:59,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:39:59,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 17:39:59,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35282.13 MB 2025-02-14 17:39:59,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36049.13 MB 2025-02-14 17:39:59,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:39:59,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40634.42 MB 2025-02-14 17:39:59,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41051.75 MB 2025-02-14 17:39:59,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:39:59,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36756.92 MB 2025-02-14 17:39:59,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:39:59,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:39:59,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:39:59,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36462.02 MB 2025-02-14 17:39:59,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36690.93 MB 2025-02-14 17:39:59,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 17:39:59,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41051.75 MB 2025-02-14 17:39:59,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41051.75 MB 2025-02-14 17:39:59,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:39:59,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36902.39 MB 2025-02-14 17:39:59,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:39:59,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:39:59,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 46.81 seconds 2025-02-14 17:39:59,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21306.24 MB 2025-02-14 17:39:59,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36891.41 MB 2025-02-14 17:39:59,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15585.17 MB 2025-02-14 17:39:59,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22508.73 MB 2025-02-14 17:39:59,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41051.75 MB 2025-02-14 17:39:59,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18543.02 MB 2025-02-14 17:39:59,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36902.39 MB 2025-02-14 17:39:59,704 - logging.py:328 - warning_once - WARNING - The attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.45 `position_ids` will be removed and `position_embeddings` will be mandatory. 2025-02-14 17:39:59,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:39:59,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:39:59,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 17:39:59,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:39:59,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23329.85 MB 2025-02-14 17:39:59,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26335.04 MB 2025-02-14 17:39:59,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.19 MB 2025-02-14 17:39:59,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41051.75 MB 2025-02-14 17:39:59,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41051.75 MB 2025-02-14 17:39:59,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:39:59,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26635.52 MB 2025-02-14 17:39:59,988 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 17:39:59,991 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:40:00,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:40:00,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:40:00,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 17:40:00,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:40:00,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26335.04 MB 2025-02-14 17:40:00,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34749.02 MB 2025-02-14 17:40:00,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 17:40:00,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41051.75 MB 2025-02-14 17:40:00,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45233.47 MB 2025-02-14 17:40:00,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 17:40:00,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34749.02 MB 2025-02-14 17:40:00,154 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 17:40:00,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:40:00,156 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:40:00,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:40:00,157 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:40:00,161 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:40:00,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:40:00,163 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:40:00,163 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:40:49,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:40:49,082 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:40:49,087 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:40:49,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:40:49,092 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1260, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:40:49,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:40:49,093 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1260, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:41:08,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:41:08,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:41:08,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.27 seconds 2025-02-14 17:41:08,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:08,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21748.59 MB 2025-02-14 17:41:08,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26207.66 MB 2025-02-14 17:41:08,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4459.07 MB 2025-02-14 17:41:08,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57780.73 MB 2025-02-14 17:41:08,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-14 17:41:08,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22657.63 MB 2025-02-14 17:41:08,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35070.33 MB 2025-02-14 17:41:08,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:41:08,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:41:08,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 17:41:08,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:08,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26207.66 MB 2025-02-14 17:41:08,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22328.20 MB 2025-02-14 17:41:08,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3879.46 MB 2025-02-14 17:41:08,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35123.10 MB 2025-02-14 17:41:08,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44193.28 MB 2025-02-14 17:41:08,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9070.18 MB 2025-02-14 17:41:08,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39558.25 MB 2025-02-14 17:41:10,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:41:10,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:41:10,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 17:41:10,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:10,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22328.20 MB 2025-02-14 17:41:10,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22859.04 MB 2025-02-14 17:41:10,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:41:10,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44193.28 MB 2025-02-14 17:41:10,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26480.74 MB 2025-02-14 17:41:10,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17712.55 MB 2025-02-14 17:41:10,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26838.63 MB 2025-02-14 17:41:10,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:41:10,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:41:10,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:41:10,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:10,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-14 17:41:10,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24748.57 MB 2025-02-14 17:41:10,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:41:10,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26480.74 MB 2025-02-14 17:41:10,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27424.46 MB 2025-02-14 17:41:10,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 17:41:10,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26166.00 MB 2025-02-14 17:41:10,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:41:10,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:41:10,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 17:41:10,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:10,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24748.57 MB 2025-02-14 17:41:10,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-14 17:41:10,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:41:10,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27424.46 MB 2025-02-14 17:41:10,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34030.49 MB 2025-02-14 17:41:10,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 17:41:10,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-14 17:41:10,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:41:10,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:41:10,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:41:10,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:10,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-14 17:41:10,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-14 17:41:10,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:41:10,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26480.74 MB 2025-02-14 17:41:10,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34030.49 MB 2025-02-14 17:41:10,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 17:41:10,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-14 17:41:10,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:41:10,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:41:10,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 17:41:10,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:10,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28523.97 MB 2025-02-14 17:41:10,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29290.97 MB 2025-02-14 17:41:10,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:41:10,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34030.49 MB 2025-02-14 17:41:10,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 17:41:10,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 17:41:10,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29998.76 MB 2025-02-14 17:41:10,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:41:10,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:41:10,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:41:10,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:10,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29703.86 MB 2025-02-14 17:41:10,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29931.84 MB 2025-02-14 17:41:10,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 17:41:10,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 17:41:10,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 17:41:10,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:41:10,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30161.84 MB 2025-02-14 17:41:10,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:41:10,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:41:10,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.66 seconds 2025-02-14 17:41:10,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:10,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17358.65 MB 2025-02-14 17:41:10,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30131.73 MB 2025-02-14 17:41:10,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12773.08 MB 2025-02-14 17:41:10,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57780.73 MB 2025-02-14 17:41:10,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 17:41:10,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23337.11 MB 2025-02-14 17:41:10,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30161.84 MB 2025-02-14 17:41:11,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:41:11,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:41:11,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 17:41:11,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:11,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30131.73 MB 2025-02-14 17:41:11,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22345.34 MB 2025-02-14 17:41:11,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7786.39 MB 2025-02-14 17:41:11,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 17:41:11,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 17:41:11,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:41:11,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31930.11 MB 2025-02-14 17:41:11,038 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 17:41:11,038 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:41:11,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:41:11,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:41:11,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:41:11,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:41:11,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22345.34 MB 2025-02-14 17:41:11,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30734.49 MB 2025-02-14 17:41:11,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 17:41:11,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 17:41:11,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38614.86 MB 2025-02-14 17:41:11,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 17:41:11,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30734.49 MB 2025-02-14 17:41:11,198 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 17:41:11,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:41:11,200 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:41:11,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:41:11,201 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:41:11,205 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:41:11,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:41:11,206 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:41:11,206 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:42:07,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:42:07,209 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:42:07,214 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:42:07,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:42:07,217 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1133, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:42:07,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:42:07,218 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1133, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:42:24,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:42:24,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:42:24,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.32 seconds 2025-02-14 17:42:24,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:24,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20863.63 MB 2025-02-14 17:42:24,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24873.39 MB 2025-02-14 17:42:24,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4009.75 MB 2025-02-14 17:42:24,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46957.33 MB 2025-02-14 17:42:24,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30477.91 MB 2025-02-14 17:42:24,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16479.42 MB 2025-02-14 17:42:24,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33732.39 MB 2025-02-14 17:42:24,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:42:24,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:42:24,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 17:42:24,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:24,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24873.39 MB 2025-02-14 17:42:24,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21667.96 MB 2025-02-14 17:42:24,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3205.42 MB 2025-02-14 17:42:24,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30477.91 MB 2025-02-14 17:42:24,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40221.28 MB 2025-02-14 17:42:24,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9743.37 MB 2025-02-14 17:42:24,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36006.78 MB 2025-02-14 17:42:26,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:42:26,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:42:26,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 17:42:26,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:26,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21667.96 MB 2025-02-14 17:42:26,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22198.81 MB 2025-02-14 17:42:26,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:42:26,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40221.28 MB 2025-02-14 17:42:26,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27883.73 MB 2025-02-14 17:42:26,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12337.55 MB 2025-02-14 17:42:26,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26177.35 MB 2025-02-14 17:42:26,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:42:26,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:42:26,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:42:26,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:26,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22198.81 MB 2025-02-14 17:42:26,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24088.34 MB 2025-02-14 17:42:26,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:42:26,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 17:42:26,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27883.73 MB 2025-02-14 17:42:26,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:42:26,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25505.77 MB 2025-02-14 17:42:26,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:42:26,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:42:26,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 17:42:26,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:26,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24088.34 MB 2025-02-14 17:42:26,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26330.20 MB 2025-02-14 17:42:26,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:42:26,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 17:42:26,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34017.90 MB 2025-02-14 17:42:26,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 17:42:26,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31874.48 MB 2025-02-14 17:42:26,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:42:26,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:42:26,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:42:26,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:26,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22198.81 MB 2025-02-14 17:42:26,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26330.20 MB 2025-02-14 17:42:26,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:42:26,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 17:42:26,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34017.90 MB 2025-02-14 17:42:26,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 17:42:26,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31874.48 MB 2025-02-14 17:42:26,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:42:26,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:42:26,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 17:42:26,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:26,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27863.74 MB 2025-02-14 17:42:26,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28630.74 MB 2025-02-14 17:42:26,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:42:26,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34017.90 MB 2025-02-14 17:42:26,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34433.14 MB 2025-02-14 17:42:26,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 17:42:26,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29338.53 MB 2025-02-14 17:42:26,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:42:26,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:42:26,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:42:26,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:26,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29043.63 MB 2025-02-14 17:42:26,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29271.80 MB 2025-02-14 17:42:26,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 17:42:26,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34433.14 MB 2025-02-14 17:42:26,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34433.14 MB 2025-02-14 17:42:26,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:42:26,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29454.01 MB 2025-02-14 17:42:26,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:42:26,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:42:26,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.70 seconds 2025-02-14 17:42:26,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:26,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16916.17 MB 2025-02-14 17:42:26,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29472.46 MB 2025-02-14 17:42:26,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12556.29 MB 2025-02-14 17:42:26,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46957.33 MB 2025-02-14 17:42:26,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34433.14 MB 2025-02-14 17:42:26,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12524.19 MB 2025-02-14 17:42:26,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29472.46 MB 2025-02-14 17:42:27,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:42:27,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:42:27,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:42:27,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:27,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29472.46 MB 2025-02-14 17:42:27,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21914.08 MB 2025-02-14 17:42:27,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7558.38 MB 2025-02-14 17:42:27,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34433.14 MB 2025-02-14 17:42:27,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34433.14 MB 2025-02-14 17:42:27,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:42:27,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31978.90 MB 2025-02-14 17:42:27,204 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 17:42:27,204 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 17:42:27,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:42:27,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:42:27,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:42:27,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:42:27,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21914.08 MB 2025-02-14 17:42:27,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30336.04 MB 2025-02-14 17:42:27,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 17:42:27,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34433.14 MB 2025-02-14 17:42:27,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42804.97 MB 2025-02-14 17:42:27,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 17:42:27,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.04 MB 2025-02-14 17:42:27,366 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 17:42:27,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:42:27,367 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:42:27,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:42:27,368 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:42:27,373 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:42:27,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:42:27,374 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:42:27,374 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 17:43:49,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:43:49,016 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:43:49,020 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:43:49,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:43:49,024 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1327, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:43:49,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:43:49,026 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1327, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:44:09,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:44:09,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:44:09,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.24 seconds 2025-02-14 17:44:09,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:09,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22215.45 MB 2025-02-14 17:44:09,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26911.63 MB 2025-02-14 17:44:09,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4696.18 MB 2025-02-14 17:44:09,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51176.80 MB 2025-02-14 17:44:09,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35353.79 MB 2025-02-14 17:44:09,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15823.01 MB 2025-02-14 17:44:09,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35763.69 MB 2025-02-14 17:44:09,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:44:09,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:44:09,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 17:44:09,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:09,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26911.63 MB 2025-02-14 17:44:09,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22676.51 MB 2025-02-14 17:44:09,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4235.12 MB 2025-02-14 17:44:09,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35353.79 MB 2025-02-14 17:44:09,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45745.18 MB 2025-02-14 17:44:09,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10391.39 MB 2025-02-14 17:44:09,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40908.13 MB 2025-02-14 17:44:11,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:44:11,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:44:11,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 17:44:11,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22676.51 MB 2025-02-14 17:44:11,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23207.35 MB 2025-02-14 17:44:11,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:44:11,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45745.18 MB 2025-02-14 17:44:11,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30656.17 MB 2025-02-14 17:44:11,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15089.01 MB 2025-02-14 17:44:11,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27185.90 MB 2025-02-14 17:44:11,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:44:11,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:44:11,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:44:11,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-14 17:44:11,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25096.89 MB 2025-02-14 17:44:11,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:44:11,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30656.17 MB 2025-02-14 17:44:11,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30656.17 MB 2025-02-14 17:44:11,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:44:11,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26514.31 MB 2025-02-14 17:44:11,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:44:11,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:44:11,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 17:44:11,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25096.89 MB 2025-02-14 17:44:11,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-14 17:44:11,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:44:11,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30656.17 MB 2025-02-14 17:44:11,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35374.76 MB 2025-02-14 17:44:11,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 17:44:11,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-14 17:44:11,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:44:11,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:44:11,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:44:11,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-14 17:44:11,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-14 17:44:11,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:44:11,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30656.17 MB 2025-02-14 17:44:11,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35374.76 MB 2025-02-14 17:44:11,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 17:44:11,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-14 17:44:11,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:44:11,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:44:11,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:44:11,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28872.28 MB 2025-02-14 17:44:11,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29639.29 MB 2025-02-14 17:44:11,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:44:11,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35374.76 MB 2025-02-14 17:44:11,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35792.09 MB 2025-02-14 17:44:11,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:44:11,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30347.07 MB 2025-02-14 17:44:11,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:44:11,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:44:11,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:44:11,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30052.17 MB 2025-02-14 17:44:11,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30280.13 MB 2025-02-14 17:44:11,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-14 17:44:11,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35792.09 MB 2025-02-14 17:44:11,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35792.09 MB 2025-02-14 17:44:11,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:44:11,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.67 MB 2025-02-14 17:44:11,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:44:11,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:44:11,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.68 seconds 2025-02-14 17:44:11,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17592.08 MB 2025-02-14 17:44:11,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30480.95 MB 2025-02-14 17:44:11,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12888.87 MB 2025-02-14 17:44:11,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51176.80 MB 2025-02-14 17:44:11,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35792.09 MB 2025-02-14 17:44:11,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15384.71 MB 2025-02-14 17:44:11,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.67 MB 2025-02-14 17:44:11,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:44:11,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:44:11,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 17:44:11,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30480.95 MB 2025-02-14 17:44:11,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22592.66 MB 2025-02-14 17:44:11,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7888.29 MB 2025-02-14 17:44:11,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35792.09 MB 2025-02-14 17:44:11,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35792.09 MB 2025-02-14 17:44:11,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:44:11,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32989.55 MB 2025-02-14 17:44:11,989 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 17:44:11,990 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:44:11,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:44:11,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:44:11,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:44:11,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:44:11,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22592.66 MB 2025-02-14 17:44:11,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31021.78 MB 2025-02-14 17:44:11,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 17:44:11,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35792.09 MB 2025-02-14 17:44:11,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44172.31 MB 2025-02-14 17:44:11,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 17:44:11,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31021.78 MB 2025-02-14 17:44:12,152 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 17:44:12,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:44:12,154 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:44:12,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:44:12,155 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:44:12,159 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:44:12,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:44:12,160 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:44:12,160 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:45:01,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:01,025 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:45:01,030 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:45:01,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:01,033 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1782, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:45:01,034 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:01,034 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1782, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:45:28,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:45:28,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:45:28,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.42 seconds 2025-02-14 17:45:28,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:28,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25385.97 MB 2025-02-14 17:45:28,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31692.37 MB 2025-02-14 17:45:28,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6306.40 MB 2025-02-14 17:45:28,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52552.53 MB 2025-02-14 17:45:28,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36964.40 MB 2025-02-14 17:45:28,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15588.13 MB 2025-02-14 17:45:28,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40520.46 MB 2025-02-14 17:45:28,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:45:28,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:45:28,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 17:45:28,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:28,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31692.37 MB 2025-02-14 17:45:28,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25041.91 MB 2025-02-14 17:45:28,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6650.45 MB 2025-02-14 17:45:28,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36964.40 MB 2025-02-14 17:45:28,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59414.41 MB 2025-02-14 17:45:28,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22450.01 MB 2025-02-14 17:45:28,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50148.54 MB 2025-02-14 17:45:30,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:45:30,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:45:30,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 17:45:30,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:30,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25041.91 MB 2025-02-14 17:45:30,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25572.75 MB 2025-02-14 17:45:30,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:45:30,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59414.41 MB 2025-02-14 17:45:30,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27885.83 MB 2025-02-14 17:45:30,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31528.58 MB 2025-02-14 17:45:30,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29552.34 MB 2025-02-14 17:45:30,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:45:30,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:45:30,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:45:30,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:30,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25572.75 MB 2025-02-14 17:45:30,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27462.29 MB 2025-02-14 17:45:30,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:45:30,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27885.83 MB 2025-02-14 17:45:30,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30716.99 MB 2025-02-14 17:45:30,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 17:45:30,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28879.72 MB 2025-02-14 17:45:30,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:45:30,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:45:30,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 17:45:30,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:30,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27462.29 MB 2025-02-14 17:45:30,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29704.14 MB 2025-02-14 17:45:30,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:45:30,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30716.99 MB 2025-02-14 17:45:30,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36851.15 MB 2025-02-14 17:45:30,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 17:45:30,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35248.43 MB 2025-02-14 17:45:30,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:45:30,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:45:30,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:45:30,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:30,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25572.75 MB 2025-02-14 17:45:30,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29704.14 MB 2025-02-14 17:45:30,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:45:30,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27885.83 MB 2025-02-14 17:45:30,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36851.15 MB 2025-02-14 17:45:30,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 17:45:30,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35248.43 MB 2025-02-14 17:45:30,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:45:30,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:45:30,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 17:45:30,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:30,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31237.69 MB 2025-02-14 17:45:30,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32004.69 MB 2025-02-14 17:45:30,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:45:30,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36851.15 MB 2025-02-14 17:45:30,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37268.49 MB 2025-02-14 17:45:30,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:45:30,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32712.48 MB 2025-02-14 17:45:30,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:45:30,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:45:30,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:45:30,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:30,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32417.58 MB 2025-02-14 17:45:30,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32646.22 MB 2025-02-14 17:45:30,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-14 17:45:30,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37268.49 MB 2025-02-14 17:45:30,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37268.49 MB 2025-02-14 17:45:30,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:45:30,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32851.27 MB 2025-02-14 17:45:30,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:45:30,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:45:30,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.90 seconds 2025-02-14 17:45:30,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:30,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19177.34 MB 2025-02-14 17:45:30,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32847.19 MB 2025-02-14 17:45:30,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13669.86 MB 2025-02-14 17:45:30,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52552.53 MB 2025-02-14 17:45:30,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37268.49 MB 2025-02-14 17:45:30,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15284.04 MB 2025-02-14 17:45:30,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32851.27 MB 2025-02-14 17:45:31,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:45:31,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:45:31,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:45:31,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:31,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32847.19 MB 2025-02-14 17:45:31,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24180.20 MB 2025-02-14 17:45:31,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8666.99 MB 2025-02-14 17:45:31,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37268.49 MB 2025-02-14 17:45:31,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37268.49 MB 2025-02-14 17:45:31,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:45:31,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35357.63 MB 2025-02-14 17:45:31,222 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 17:45:31,223 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:45:31,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:45:31,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:45:31,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:45:31,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:31,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24180.20 MB 2025-02-14 17:45:31,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32615.05 MB 2025-02-14 17:45:31,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 17:45:31,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37268.49 MB 2025-02-14 17:45:31,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45655.00 MB 2025-02-14 17:45:31,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 17:45:31,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32615.05 MB 2025-02-14 17:45:31,388 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 17:45:31,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:31,390 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:45:31,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:31,391 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:45:31,395 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:45:31,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:31,396 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:45:31,396 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:45:41,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:41,451 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:45:41,458 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:45:41,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:41,464 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1130, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:45:41,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:45:41,466 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1130, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:45:59,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:45:59,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:45:59,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.61 seconds 2025-02-14 17:45:59,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:59,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20842.73 MB 2025-02-14 17:45:59,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24841.73 MB 2025-02-14 17:45:59,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3999.01 MB 2025-02-14 17:45:59,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58233.72 MB 2025-02-14 17:45:59,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26491.22 MB 2025-02-14 17:45:59,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31742.49 MB 2025-02-14 17:45:59,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33711.48 MB 2025-02-14 17:45:59,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:45:59,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:45:59,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 17:45:59,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:45:59,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24841.73 MB 2025-02-14 17:45:59,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21653.42 MB 2025-02-14 17:45:59,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3188.32 MB 2025-02-14 17:45:59,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26491.22 MB 2025-02-14 17:45:59,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45929.73 MB 2025-02-14 17:45:59,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19438.50 MB 2025-02-14 17:45:59,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36990.80 MB 2025-02-14 17:46:01,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:46:01,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:46:01,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 17:46:01,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21653.42 MB 2025-02-14 17:46:01,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22184.26 MB 2025-02-14 17:46:01,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:46:01,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45929.73 MB 2025-02-14 17:46:01,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29324.48 MB 2025-02-14 17:46:01,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16605.25 MB 2025-02-14 17:46:01,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26162.81 MB 2025-02-14 17:46:01,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:46:01,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:46:01,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:01,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22184.26 MB 2025-02-14 17:46:01,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24073.79 MB 2025-02-14 17:46:01,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:46:01,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29324.48 MB 2025-02-14 17:46:01,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29324.48 MB 2025-02-14 17:46:01,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:01,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25491.22 MB 2025-02-14 17:46:01,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:46:01,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:46:01,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:46:01,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24073.79 MB 2025-02-14 17:46:01,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26315.65 MB 2025-02-14 17:46:01,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:46:01,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29324.48 MB 2025-02-14 17:46:01,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-14 17:46:01,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 17:46:01,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31859.93 MB 2025-02-14 17:46:01,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:46:01,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:46:01,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:46:01,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22184.26 MB 2025-02-14 17:46:01,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26315.65 MB 2025-02-14 17:46:01,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:46:01,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29324.48 MB 2025-02-14 17:46:01,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-14 17:46:01,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 17:46:01,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31859.93 MB 2025-02-14 17:46:01,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:46:01,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:46:01,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:46:01,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27849.19 MB 2025-02-14 17:46:01,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28616.19 MB 2025-02-14 17:46:01,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:46:01,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33571.21 MB 2025-02-14 17:46:01,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33988.54 MB 2025-02-14 17:46:01,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:46:01,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29323.98 MB 2025-02-14 17:46:01,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:46:01,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:46:01,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:46:01,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29029.08 MB 2025-02-14 17:46:01,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29256.80 MB 2025-02-14 17:46:01,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.72 MB 2025-02-14 17:46:01,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33988.54 MB 2025-02-14 17:46:01,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33988.54 MB 2025-02-14 17:46:01,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:01,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29487.92 MB 2025-02-14 17:46:01,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:46:01,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:46:01,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.06 seconds 2025-02-14 17:46:01,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16905.72 MB 2025-02-14 17:46:01,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29457.60 MB 2025-02-14 17:46:01,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12551.89 MB 2025-02-14 17:46:01,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58233.72 MB 2025-02-14 17:46:01,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33988.54 MB 2025-02-14 17:46:01,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24245.17 MB 2025-02-14 17:46:01,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29487.92 MB 2025-02-14 17:46:01,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:46:01,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:46:01,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:46:01,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29457.60 MB 2025-02-14 17:46:01,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21905.92 MB 2025-02-14 17:46:01,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7551.69 MB 2025-02-14 17:46:01,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33988.54 MB 2025-02-14 17:46:01,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33988.54 MB 2025-02-14 17:46:01,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:01,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31965.89 MB 2025-02-14 17:46:01,822 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 17:46:01,823 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:46:01,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:46:01,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:46:01,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:46:01,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:01,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21905.92 MB 2025-02-14 17:46:01,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30333.25 MB 2025-02-14 17:46:01,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 17:46:01,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33988.54 MB 2025-02-14 17:46:01,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42368.76 MB 2025-02-14 17:46:01,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 17:46:01,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30333.25 MB 2025-02-14 17:46:02,050 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 17:46:02,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:02,053 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:46:02,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:02,055 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:46:02,061 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:46:02,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:02,063 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:46:02,063 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:46:15,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:15,473 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:46:15,478 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:46:15,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:15,481 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:46:15,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:15,482 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:46:18,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:46:18,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:46:18,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.91 seconds 2025-02-14 17:46:18,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:18,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-14 17:46:18,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-14 17:46:18,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-14 17:46:18,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50748.98 MB 2025-02-14 17:46:18,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 17:46:18,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27944.55 MB 2025-02-14 17:46:18,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.15 MB 2025-02-14 17:46:18,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:46:18,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:46:18,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:18,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:18,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-14 17:46:18,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15241.88 MB 2025-02-14 17:46:18,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.85 MB 2025-02-14 17:46:18,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 17:46:18,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 17:46:18,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:18,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17570.98 MB 2025-02-14 17:46:19,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:46:19,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:46:19,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-14 17:46:19,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15241.88 MB 2025-02-14 17:46:19,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15488.72 MB 2025-02-14 17:46:19,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-14 17:46:19,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 17:46:19,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 17:46:19,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:19,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19411.53 MB 2025-02-14 17:46:19,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:46:19,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:46:19,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:19,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-14 17:46:19,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16367.14 MB 2025-02-14 17:46:19,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-14 17:46:19,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 17:46:19,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 17:46:19,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:19,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17026.25 MB 2025-02-14 17:46:19,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:46:19,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:46:19,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 17:46:19,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16367.14 MB 2025-02-14 17:46:19,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-14 17:46:19,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-14 17:46:19,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 17:46:19,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 17:46:19,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:19,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.70 MB 2025-02-14 17:46:19,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:46:19,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:46:19,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 17:46:19,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-14 17:46:19,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-14 17:46:19,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-14 17:46:19,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 17:46:19,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 17:46:19,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:19,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.70 MB 2025-02-14 17:46:19,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:46:19,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:46:19,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 17:46:19,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18122.74 MB 2025-02-14 17:46:19,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18479.39 MB 2025-02-14 17:46:19,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.66 MB 2025-02-14 17:46:19,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 17:46:19,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22997.37 MB 2025-02-14 17:46:19,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 192.94 MB 2025-02-14 17:46:19,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18812.86 MB 2025-02-14 17:46:19,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:46:19,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:46:19,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:19,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18671.39 MB 2025-02-14 17:46:19,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18875.10 MB 2025-02-14 17:46:19,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.71 MB 2025-02-14 17:46:19,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22997.37 MB 2025-02-14 17:46:19,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23001.56 MB 2025-02-14 17:46:19,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 17:46:19,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18902.49 MB 2025-02-14 17:46:19,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:46:19,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:46:19,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.02 seconds 2025-02-14 17:46:19,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-14 17:46:19,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19076.08 MB 2025-02-14 17:46:19,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5459.33 MB 2025-02-14 17:46:19,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50748.98 MB 2025-02-14 17:46:19,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23001.56 MB 2025-02-14 17:46:19,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27747.42 MB 2025-02-14 17:46:19,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19076.08 MB 2025-02-14 17:46:19,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:46:19,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:46:19,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:46:19,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19076.08 MB 2025-02-14 17:46:19,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17609.58 MB 2025-02-14 17:46:19,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1466.49 MB 2025-02-14 17:46:19,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23001.56 MB 2025-02-14 17:46:19,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23001.56 MB 2025-02-14 17:46:19,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:19,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19076.08 MB 2025-02-14 17:46:19,791 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 17:46:19,791 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:46:19,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:46:19,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:46:19,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:46:19,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:19,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17609.58 MB 2025-02-14 17:46:19,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26044.43 MB 2025-02-14 17:46:19,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 17:46:19,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23001.56 MB 2025-02-14 17:46:19,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31388.07 MB 2025-02-14 17:46:19,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 17:46:19,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26044.43 MB 2025-02-14 17:46:19,953 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 17:46:19,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:19,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:46:19,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:19,955 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:46:19,960 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:46:19,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:19,961 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:46:19,961 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:46:36,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:36,221 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:46:36,229 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:46:36,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:36,235 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 234, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:46:36,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:36,237 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 234, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:46:39,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:46:39,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:46:39,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.66 seconds 2025-02-14 17:46:39,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:39,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.26 MB 2025-02-14 17:46:39,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15427.37 MB 2025-02-14 17:46:39,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 828.11 MB 2025-02-14 17:46:39,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43966.79 MB 2025-02-14 17:46:39,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21676.16 MB 2025-02-14 17:46:39,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22290.63 MB 2025-02-14 17:46:39,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24297.12 MB 2025-02-14 17:46:39,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:46:39,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:46:39,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:39,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:39,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15427.37 MB 2025-02-14 17:46:39,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15744.25 MB 2025-02-14 17:46:39,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 316.88 MB 2025-02-14 17:46:39,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21676.16 MB 2025-02-14 17:46:39,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21676.16 MB 2025-02-14 17:46:39,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:39,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18580.98 MB 2025-02-14 17:46:40,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:46:40,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:46:40,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-14 17:46:40,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:40,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15744.25 MB 2025-02-14 17:46:40,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16038.86 MB 2025-02-14 17:46:40,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-14 17:46:40,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21676.16 MB 2025-02-14 17:46:40,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21204.30 MB 2025-02-14 17:46:40,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 17:46:40,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19998.83 MB 2025-02-14 17:46:41,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:46:41,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:46:41,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:41,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16038.86 MB 2025-02-14 17:46:41,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17087.30 MB 2025-02-14 17:46:41,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.44 MB 2025-02-14 17:46:41,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21204.30 MB 2025-02-14 17:46:41,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21204.30 MB 2025-02-14 17:46:41,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:41,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17873.98 MB 2025-02-14 17:46:41,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:46:41,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:46:41,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 17:46:41,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17087.30 MB 2025-02-14 17:46:41,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18331.56 MB 2025-02-14 17:46:41,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1244.26 MB 2025-02-14 17:46:41,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21204.30 MB 2025-02-14 17:46:41,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23039.31 MB 2025-02-14 17:46:41,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1835.01 MB 2025-02-14 17:46:41,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21410.44 MB 2025-02-14 17:46:41,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:46:41,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:46:41,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 17:46:41,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16038.86 MB 2025-02-14 17:46:41,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18331.56 MB 2025-02-14 17:46:41,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2292.70 MB 2025-02-14 17:46:41,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21204.30 MB 2025-02-14 17:46:41,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23039.31 MB 2025-02-14 17:46:41,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1835.01 MB 2025-02-14 17:46:41,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21410.44 MB 2025-02-14 17:46:41,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:46:41,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:46:41,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 17:46:41,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19182.67 MB 2025-02-14 17:46:41,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19608.62 MB 2025-02-14 17:46:41,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.95 MB 2025-02-14 17:46:41,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23039.31 MB 2025-02-14 17:46:41,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23272.10 MB 2025-02-14 17:46:41,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 232.78 MB 2025-02-14 17:46:41,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20001.54 MB 2025-02-14 17:46:41,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:46:41,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:46:41,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:41,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19837.78 MB 2025-02-14 17:46:41,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20057.70 MB 2025-02-14 17:46:41,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.92 MB 2025-02-14 17:46:41,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23272.10 MB 2025-02-14 17:46:41,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23272.10 MB 2025-02-14 17:46:41,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:41,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20125.76 MB 2025-02-14 17:46:41,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:46:41,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:46:41,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.98 seconds 2025-02-14 17:46:41,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13783.98 MB 2025-02-14 17:46:41,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20258.43 MB 2025-02-14 17:46:41,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6474.44 MB 2025-02-14 17:46:41,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43966.79 MB 2025-02-14 17:46:41,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23272.10 MB 2025-02-14 17:46:41,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20694.70 MB 2025-02-14 17:46:41,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20258.43 MB 2025-02-14 17:46:41,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:46:41,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:46:41,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:46:41,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14934.39 MB 2025-02-14 17:46:41,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17943.26 MB 2025-02-14 17:46:41,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.87 MB 2025-02-14 17:46:41,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23272.10 MB 2025-02-14 17:46:41,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23272.10 MB 2025-02-14 17:46:41,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:41,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18244.12 MB 2025-02-14 17:46:41,509 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 17:46:41,510 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:46:41,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:46:41,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:46:41,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:46:41,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:41,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17943.26 MB 2025-02-14 17:46:41,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26368.22 MB 2025-02-14 17:46:41,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 17:46:41,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23272.10 MB 2025-02-14 17:46:41,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31648.12 MB 2025-02-14 17:46:41,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 17:46:41,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26368.22 MB 2025-02-14 17:46:41,671 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 17:46:41,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:41,673 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:46:41,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:41,674 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:46:41,678 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:46:41,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:41,680 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:46:41,680 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:46:52,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:52,529 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:46:52,537 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:46:52,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:52,543 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 287, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:46:52,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:52,545 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 287, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:46:57,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:46:57,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:46:57,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.53 seconds 2025-02-14 17:46:57,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:57,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14968.57 MB 2025-02-14 17:46:57,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15984.25 MB 2025-02-14 17:46:57,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1015.68 MB 2025-02-14 17:46:57,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40024.15 MB 2025-02-14 17:46:57,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22777.17 MB 2025-02-14 17:46:57,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17246.98 MB 2025-02-14 17:46:57,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24892.92 MB 2025-02-14 17:46:57,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:46:57,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:46:57,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:46:57,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:57,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15984.25 MB 2025-02-14 17:46:57,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16328.79 MB 2025-02-14 17:46:57,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 344.54 MB 2025-02-14 17:46:57,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22777.17 MB 2025-02-14 17:46:57,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22777.17 MB 2025-02-14 17:46:57,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:46:57,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19773.58 MB 2025-02-14 17:46:58,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:46:58,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:46:58,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.29 seconds 2025-02-14 17:46:58,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16328.79 MB 2025-02-14 17:46:58,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16681.80 MB 2025-02-14 17:46:58,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.01 MB 2025-02-14 17:46:58,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22777.17 MB 2025-02-14 17:46:58,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20103.30 MB 2025-02-14 17:46:58,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2673.87 MB 2025-02-14 17:46:58,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20669.35 MB 2025-02-14 17:46:58,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:46:58,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:46:58,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:58,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16681.80 MB 2025-02-14 17:46:58,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.08 MB 2025-02-14 17:46:58,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1256.28 MB 2025-02-14 17:46:58,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20103.30 MB 2025-02-14 17:46:58,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20732.44 MB 2025-02-14 17:46:58,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 629.15 MB 2025-02-14 17:46:58,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18880.68 MB 2025-02-14 17:46:58,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:46:58,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:46:58,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 17:46:58,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.08 MB 2025-02-14 17:46:58,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19429.09 MB 2025-02-14 17:46:58,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1491.01 MB 2025-02-14 17:46:58,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20732.44 MB 2025-02-14 17:46:58,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24507.32 MB 2025-02-14 17:46:58,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 17:46:58,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23116.80 MB 2025-02-14 17:46:58,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:46:58,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:46:58,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 17:46:58,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16681.80 MB 2025-02-14 17:46:58,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19429.09 MB 2025-02-14 17:46:58,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2747.29 MB 2025-02-14 17:46:58,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20103.30 MB 2025-02-14 17:46:58,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24507.32 MB 2025-02-14 17:46:58,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4404.02 MB 2025-02-14 17:46:58,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23116.80 MB 2025-02-14 17:46:58,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:46:58,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:46:58,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 17:46:58,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20448.90 MB 2025-02-14 17:46:58,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20959.74 MB 2025-02-14 17:46:58,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 510.84 MB 2025-02-14 17:46:58,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24507.32 MB 2025-02-14 17:46:58,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24786.24 MB 2025-02-14 17:46:58,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 278.92 MB 2025-02-14 17:46:58,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21430.42 MB 2025-02-14 17:46:58,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:46:58,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:46:58,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:46:58,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21234.32 MB 2025-02-14 17:46:58,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21438.80 MB 2025-02-14 17:46:58,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.48 MB 2025-02-14 17:46:58,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24786.24 MB 2025-02-14 17:46:58,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24790.43 MB 2025-02-14 17:46:58,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 17:46:58,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21502.74 MB 2025-02-14 17:46:58,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:46:58,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:46:58,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.12 seconds 2025-02-14 17:46:58,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13968.64 MB 2025-02-14 17:46:58,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21639.80 MB 2025-02-14 17:46:58,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7671.16 MB 2025-02-14 17:46:58,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40024.15 MB 2025-02-14 17:46:58,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24790.43 MB 2025-02-14 17:46:58,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15233.71 MB 2025-02-14 17:46:58,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21639.80 MB 2025-02-14 17:46:58,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:46:58,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:46:58,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:46:58,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21639.80 MB 2025-02-14 17:46:58,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24652.73 MB 2025-02-14 17:46:58,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.93 MB 2025-02-14 17:46:58,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24790.43 MB 2025-02-14 17:46:58,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26132.61 MB 2025-02-14 17:46:58,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1342.18 MB 2025-02-14 17:46:58,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24954.29 MB 2025-02-14 17:46:58,953 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 17:46:58,953 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:46:58,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:46:58,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:46:58,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:46:58,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:46:58,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18339.76 MB 2025-02-14 17:46:58,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26775.35 MB 2025-02-14 17:46:58,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 17:46:58,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26132.61 MB 2025-02-14 17:46:58,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34521.22 MB 2025-02-14 17:46:58,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 17:46:58,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26775.35 MB 2025-02-14 17:46:59,116 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 17:46:59,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:59,117 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:46:59,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:59,118 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:46:59,123 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:46:59,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:46:59,124 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:46:59,124 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:47:36,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:47:36,681 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:47:36,685 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:47:36,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:47:36,689 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 188, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:47:36,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:47:36,690 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 188, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:47:39,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:47:39,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:47:39,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.89 seconds 2025-02-14 17:47:39,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:39,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14278.72 MB 2025-02-14 17:47:39,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14944.04 MB 2025-02-14 17:47:39,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 665.32 MB 2025-02-14 17:47:39,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42909.83 MB 2025-02-14 17:47:39,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 17:47:39,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19845.35 MB 2025-02-14 17:47:39,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23750.09 MB 2025-02-14 17:47:39,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:47:39,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:47:39,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:47:39,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:39,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14944.04 MB 2025-02-14 17:47:39,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15266.39 MB 2025-02-14 17:47:39,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.35 MB 2025-02-14 17:47:39,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 17:47:39,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 17:47:39,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:47:39,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17627.23 MB 2025-02-14 17:47:40,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:47:40,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:47:40,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-14 17:47:40,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15266.39 MB 2025-02-14 17:47:40,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15515.88 MB 2025-02-14 17:47:40,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.50 MB 2025-02-14 17:47:40,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 17:47:40,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22540.19 MB 2025-02-14 17:47:40,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -524.29 MB 2025-02-14 17:47:40,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19436.04 MB 2025-02-14 17:47:40,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:47:40,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:47:40,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:47:40,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15515.82 MB 2025-02-14 17:47:40,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16403.68 MB 2025-02-14 17:47:40,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 887.87 MB 2025-02-14 17:47:40,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22540.19 MB 2025-02-14 17:47:40,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22540.19 MB 2025-02-14 17:47:40,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:47:40,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17069.88 MB 2025-02-14 17:47:40,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:47:40,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:47:40,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 17:47:40,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16403.68 MB 2025-02-14 17:47:40,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17457.39 MB 2025-02-14 17:47:40,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1053.71 MB 2025-02-14 17:47:40,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22540.19 MB 2025-02-14 17:47:40,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22540.19 MB 2025-02-14 17:47:40,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:47:40,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20063.17 MB 2025-02-14 17:47:40,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:47:40,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:47:40,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 17:47:40,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15515.82 MB 2025-02-14 17:47:40,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17457.39 MB 2025-02-14 17:47:40,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1941.57 MB 2025-02-14 17:47:40,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22540.19 MB 2025-02-14 17:47:40,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22540.19 MB 2025-02-14 17:47:40,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:47:40,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20063.17 MB 2025-02-14 17:47:40,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:47:40,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:47:40,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 17:47:40,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18178.16 MB 2025-02-14 17:47:40,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18538.65 MB 2025-02-14 17:47:40,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 360.49 MB 2025-02-14 17:47:40,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22540.19 MB 2025-02-14 17:47:40,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22737.32 MB 2025-02-14 17:47:40,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-14 17:47:40,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18876.09 MB 2025-02-14 17:47:40,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:47:40,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:47:40,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:47:40,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18732.71 MB 2025-02-14 17:47:40,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18945.06 MB 2025-02-14 17:47:40,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.35 MB 2025-02-14 17:47:40,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22737.32 MB 2025-02-14 17:47:40,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22737.32 MB 2025-02-14 17:47:40,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:47:40,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18988.61 MB 2025-02-14 17:47:40,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:47:40,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:47:40,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.03 seconds 2025-02-14 17:47:40,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13623.71 MB 2025-02-14 17:47:40,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19146.13 MB 2025-02-14 17:47:40,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5522.42 MB 2025-02-14 17:47:40,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42909.83 MB 2025-02-14 17:47:40,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22737.32 MB 2025-02-14 17:47:40,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20172.51 MB 2025-02-14 17:47:40,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19146.13 MB 2025-02-14 17:47:40,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:47:40,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:47:40,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 17:47:40,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:40,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19146.13 MB 2025-02-14 17:47:40,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17627.61 MB 2025-02-14 17:47:40,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1518.52 MB 2025-02-14 17:47:40,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22737.32 MB 2025-02-14 17:47:40,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22737.32 MB 2025-02-14 17:47:40,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:47:40,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19146.14 MB 2025-02-14 17:47:41,009 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 17:47:41,010 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:47:41,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:47:41,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:47:41,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:47:41,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:47:41,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17627.61 MB 2025-02-14 17:47:41,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26066.64 MB 2025-02-14 17:47:41,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 17:47:41,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22737.32 MB 2025-02-14 17:47:41,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31128.03 MB 2025-02-14 17:47:41,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 17:47:41,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26066.64 MB 2025-02-14 17:47:41,172 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 17:47:41,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:47:41,174 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:47:41,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:47:41,175 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:47:41,179 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:47:41,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:47:41,181 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:47:41,181 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 17:48:04,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:48:04,904 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:48:04,911 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:48:04,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:48:04,918 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 944, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:48:04,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:48:04,920 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 944, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:48:19,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:48:19,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:48:19,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.62 seconds 2025-02-14 17:48:19,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:19,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19546.65 MB 2025-02-14 17:48:19,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22887.41 MB 2025-02-14 17:48:19,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3340.76 MB 2025-02-14 17:48:19,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43713.04 MB 2025-02-14 17:48:19,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28923.92 MB 2025-02-14 17:48:19,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14789.12 MB 2025-02-14 17:48:19,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31735.93 MB 2025-02-14 17:48:19,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:48:19,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:48:19,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 17:48:19,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:19,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22887.41 MB 2025-02-14 17:48:19,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20685.41 MB 2025-02-14 17:48:19,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2202.00 MB 2025-02-14 17:48:19,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28923.92 MB 2025-02-14 17:48:19,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35475.42 MB 2025-02-14 17:48:19,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6551.50 MB 2025-02-14 17:48:19,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32304.75 MB 2025-02-14 17:48:21,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:48:21,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:48:21,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 17:48:21,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:21,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20685.41 MB 2025-02-14 17:48:21,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21216.25 MB 2025-02-14 17:48:21,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:48:21,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35475.42 MB 2025-02-14 17:48:21,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26998.73 MB 2025-02-14 17:48:21,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8476.69 MB 2025-02-14 17:48:21,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25194.80 MB 2025-02-14 17:48:21,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:48:21,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:48:21,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:48:21,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:21,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21216.25 MB 2025-02-14 17:48:21,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23105.79 MB 2025-02-14 17:48:21,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:48:21,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26998.73 MB 2025-02-14 17:48:21,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26998.73 MB 2025-02-14 17:48:21,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:48:21,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24523.22 MB 2025-02-14 17:48:21,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:48:21,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:48:21,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:48:21,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:21,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23105.79 MB 2025-02-14 17:48:21,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25347.64 MB 2025-02-14 17:48:21,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:48:21,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26998.73 MB 2025-02-14 17:48:21,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32661.05 MB 2025-02-14 17:48:21,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 17:48:21,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30891.93 MB 2025-02-14 17:48:21,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:48:21,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:48:21,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:48:21,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:21,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21216.25 MB 2025-02-14 17:48:21,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25347.64 MB 2025-02-14 17:48:21,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:48:21,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26998.73 MB 2025-02-14 17:48:21,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32661.05 MB 2025-02-14 17:48:21,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 17:48:21,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30891.93 MB 2025-02-14 17:48:21,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:48:21,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:48:21,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:48:21,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:21,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26881.19 MB 2025-02-14 17:48:21,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27648.19 MB 2025-02-14 17:48:21,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:48:21,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32661.05 MB 2025-02-14 17:48:21,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-14 17:48:21,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:48:21,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28355.98 MB 2025-02-14 17:48:21,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:48:21,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:48:21,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:48:21,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:21,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28061.08 MB 2025-02-14 17:48:21,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28290.01 MB 2025-02-14 17:48:21,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-14 17:48:21,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33078.38 MB 2025-02-14 17:48:21,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-14 17:48:21,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:48:21,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28498.96 MB 2025-02-14 17:48:21,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:48:21,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:48:21,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.00 seconds 2025-02-14 17:48:21,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:21,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16257.68 MB 2025-02-14 17:48:21,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28491.09 MB 2025-02-14 17:48:21,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12233.41 MB 2025-02-14 17:48:21,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43713.04 MB 2025-02-14 17:48:21,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-14 17:48:21,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10634.66 MB 2025-02-14 17:48:21,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28498.96 MB 2025-02-14 17:48:22,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:48:22,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:48:22,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:48:22,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:22,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28491.09 MB 2025-02-14 17:48:22,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21262.39 MB 2025-02-14 17:48:22,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7228.70 MB 2025-02-14 17:48:22,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33078.38 MB 2025-02-14 17:48:22,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-14 17:48:22,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:48:22,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31003.07 MB 2025-02-14 17:48:22,212 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 17:48:22,212 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 17:48:22,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:48:22,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:48:22,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:48:22,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:48:22,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21262.39 MB 2025-02-14 17:48:22,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29701.41 MB 2025-02-14 17:48:22,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 17:48:22,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33078.38 MB 2025-02-14 17:48:22,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41469.08 MB 2025-02-14 17:48:22,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 17:48:22,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29701.41 MB 2025-02-14 17:48:22,377 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 17:48:22,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:48:22,379 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:48:22,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:48:22,380 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:48:22,384 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:48:22,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:48:22,385 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:48:22,385 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 17:49:20,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:20,343 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:49:20,348 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:49:20,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:20,353 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 420, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:49:20,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:20,354 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 420, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:49:26,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:49:26,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:49:26,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.49 seconds 2025-02-14 17:49:26,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:26,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15895.33 MB 2025-02-14 17:49:26,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17381.69 MB 2025-02-14 17:49:26,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1486.36 MB 2025-02-14 17:49:26,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54054.09 MB 2025-02-14 17:49:26,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25163.73 MB 2025-02-14 17:49:26,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28890.37 MB 2025-02-14 17:49:26,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26272.67 MB 2025-02-14 17:49:26,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:49:26,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:49:26,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 17:49:26,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:26,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17381.69 MB 2025-02-14 17:49:26,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17961.30 MB 2025-02-14 17:49:26,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 579.61 MB 2025-02-14 17:49:26,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25163.73 MB 2025-02-14 17:49:26,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27994.88 MB 2025-02-14 17:49:26,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 17:49:26,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24465.83 MB 2025-02-14 17:49:28,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:49:28,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:49:28,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.89 seconds 2025-02-14 17:49:28,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:28,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17961.30 MB 2025-02-14 17:49:28,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18492.14 MB 2025-02-14 17:49:28,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:49:28,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27994.88 MB 2025-02-14 17:49:28,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25635.59 MB 2025-02-14 17:49:28,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2359.30 MB 2025-02-14 17:49:28,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22470.69 MB 2025-02-14 17:49:28,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:49:28,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:49:28,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:49:28,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:28,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18492.14 MB 2025-02-14 17:49:28,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20381.68 MB 2025-02-14 17:49:28,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:49:28,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25635.59 MB 2025-02-14 17:49:28,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25635.59 MB 2025-02-14 17:49:28,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:49:28,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21799.10 MB 2025-02-14 17:49:28,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:49:28,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:49:28,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 17:49:28,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:28,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20381.68 MB 2025-02-14 17:49:28,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22623.53 MB 2025-02-14 17:49:28,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:49:28,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25635.59 MB 2025-02-14 17:49:28,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30826.04 MB 2025-02-14 17:49:28,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 17:49:28,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28167.81 MB 2025-02-14 17:49:28,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:49:28,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:49:28,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:49:28,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:28,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18492.14 MB 2025-02-14 17:49:28,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22623.53 MB 2025-02-14 17:49:28,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:49:28,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25635.59 MB 2025-02-14 17:49:28,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30826.04 MB 2025-02-14 17:49:28,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 17:49:28,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28167.81 MB 2025-02-14 17:49:29,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:49:29,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:49:29,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:49:29,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:29,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24157.07 MB 2025-02-14 17:49:29,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24924.08 MB 2025-02-14 17:49:29,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:49:29,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30826.04 MB 2025-02-14 17:49:29,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31243.37 MB 2025-02-14 17:49:29,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:49:29,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25631.86 MB 2025-02-14 17:49:29,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:49:29,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:49:29,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:49:29,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:29,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25336.96 MB 2025-02-14 17:49:29,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25565.00 MB 2025-02-14 17:49:29,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.04 MB 2025-02-14 17:49:29,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31243.37 MB 2025-02-14 17:49:29,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31243.37 MB 2025-02-14 17:49:29,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:49:29,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25789.33 MB 2025-02-14 17:49:29,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:49:29,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:49:29,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.82 seconds 2025-02-14 17:49:29,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:29,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14432.02 MB 2025-02-14 17:49:29,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25765.24 MB 2025-02-14 17:49:29,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11333.22 MB 2025-02-14 17:49:29,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54054.09 MB 2025-02-14 17:49:29,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31243.37 MB 2025-02-14 17:49:29,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22810.72 MB 2025-02-14 17:49:29,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25789.33 MB 2025-02-14 17:49:29,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:49:29,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:49:29,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 17:49:29,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:29,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25765.24 MB 2025-02-14 17:49:29,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19424.15 MB 2025-02-14 17:49:29,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6341.09 MB 2025-02-14 17:49:29,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31243.37 MB 2025-02-14 17:49:29,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31243.37 MB 2025-02-14 17:49:29,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:49:29,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28266.91 MB 2025-02-14 17:49:29,462 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 17:49:29,463 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:49:29,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:49:29,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:49:29,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:49:29,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:49:29,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19424.15 MB 2025-02-14 17:49:29,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27829.23 MB 2025-02-14 17:49:29,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-14 17:49:29,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31243.37 MB 2025-02-14 17:49:29,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39598.42 MB 2025-02-14 17:49:29,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 17:49:29,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27829.23 MB 2025-02-14 17:49:29,626 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 17:49:29,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:29,628 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:49:29,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:29,629 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:49:29,633 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:49:29,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:29,634 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:49:29,634 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:49:47,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:47,113 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:49:47,120 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:49:47,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:47,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1510, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:49:47,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:49:47,129 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1510, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:50:10,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:50:10,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:50:10,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.34 seconds 2025-02-14 17:50:10,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:10,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23490.63 MB 2025-02-14 17:50:10,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28834.43 MB 2025-02-14 17:50:10,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5343.81 MB 2025-02-14 17:50:10,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47953.48 MB 2025-02-14 17:50:10,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35089.55 MB 2025-02-14 17:50:10,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12863.93 MB 2025-02-14 17:50:10,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37719.15 MB 2025-02-14 17:50:10,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:50:10,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:50:10,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 17:50:10,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:10,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28834.43 MB 2025-02-14 17:50:10,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23627.87 MB 2025-02-14 17:50:10,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5206.56 MB 2025-02-14 17:50:10,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35089.55 MB 2025-02-14 17:50:10,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42702.21 MB 2025-02-14 17:50:10,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7612.66 MB 2025-02-14 17:50:10,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39248.18 MB 2025-02-14 17:50:12,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:50:12,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:50:12,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 17:50:12,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:12,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23627.87 MB 2025-02-14 17:50:12,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24158.71 MB 2025-02-14 17:50:12,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:50:12,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42702.21 MB 2025-02-14 17:50:12,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27883.73 MB 2025-02-14 17:50:12,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14818.48 MB 2025-02-14 17:50:12,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28138.27 MB 2025-02-14 17:50:12,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:50:12,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:50:12,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:50:12,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:12,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24158.71 MB 2025-02-14 17:50:12,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.25 MB 2025-02-14 17:50:12,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:50:12,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 17:50:12,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29771.17 MB 2025-02-14 17:50:12,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 17:50:12,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27465.67 MB 2025-02-14 17:50:12,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:50:12,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:50:12,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:50:12,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:12,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.25 MB 2025-02-14 17:50:12,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28290.10 MB 2025-02-14 17:50:12,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:50:12,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29771.17 MB 2025-02-14 17:50:12,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35905.34 MB 2025-02-14 17:50:12,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 17:50:12,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33834.38 MB 2025-02-14 17:50:12,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:50:12,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:50:12,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:50:12,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:12,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24158.71 MB 2025-02-14 17:50:12,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28290.10 MB 2025-02-14 17:50:12,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:50:12,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 17:50:12,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35905.34 MB 2025-02-14 17:50:12,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 17:50:12,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33834.38 MB 2025-02-14 17:50:12,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:50:12,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:50:12,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:50:12,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:12,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29823.64 MB 2025-02-14 17:50:12,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30590.65 MB 2025-02-14 17:50:12,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:50:12,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35905.34 MB 2025-02-14 17:50:12,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 17:50:12,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:50:12,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31298.43 MB 2025-02-14 17:50:12,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:50:12,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:50:12,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:50:12,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:12,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31003.53 MB 2025-02-14 17:50:12,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31233.02 MB 2025-02-14 17:50:12,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.49 MB 2025-02-14 17:50:12,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36322.67 MB 2025-02-14 17:50:12,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 17:50:12,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:12,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31436.31 MB 2025-02-14 17:50:12,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:50:12,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:50:12,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.75 seconds 2025-02-14 17:50:12,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:12,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18229.67 MB 2025-02-14 17:50:12,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31434.10 MB 2025-02-14 17:50:12,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13204.43 MB 2025-02-14 17:50:12,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47953.48 MB 2025-02-14 17:50:12,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 17:50:12,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11630.80 MB 2025-02-14 17:50:12,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31436.31 MB 2025-02-14 17:50:13,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:50:13,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:50:13,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:50:13,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:13,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31434.10 MB 2025-02-14 17:50:13,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23234.06 MB 2025-02-14 17:50:13,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8200.04 MB 2025-02-14 17:50:13,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36322.67 MB 2025-02-14 17:50:13,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 17:50:13,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:13,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33945.76 MB 2025-02-14 17:50:13,166 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 17:50:13,166 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:50:13,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:50:13,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:50:13,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:50:13,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:13,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23234.06 MB 2025-02-14 17:50:13,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31673.08 MB 2025-02-14 17:50:13,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 17:50:13,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36322.67 MB 2025-02-14 17:50:13,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44713.38 MB 2025-02-14 17:50:13,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 17:50:13,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31673.08 MB 2025-02-14 17:50:13,332 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 17:50:13,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:13,334 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:50:13,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:13,335 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:50:13,340 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:50:13,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:13,341 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:50:13,341 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:50:28,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:28,897 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:50:28,902 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:50:28,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:28,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 324, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:50:28,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:28,906 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 324, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:50:33,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:50:33,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:50:33,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.04 seconds 2025-02-14 17:50:33,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:33,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15226.39 MB 2025-02-14 17:50:33,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16373.01 MB 2025-02-14 17:50:33,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1146.62 MB 2025-02-14 17:50:33,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57298.39 MB 2025-02-14 17:50:33,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28397.54 MB 2025-02-14 17:50:33,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28900.85 MB 2025-02-14 17:50:33,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25377.24 MB 2025-02-14 17:50:33,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:50:33,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:50:33,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:50:33,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:33,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16373.01 MB 2025-02-14 17:50:33,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16844.20 MB 2025-02-14 17:50:33,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 471.19 MB 2025-02-14 17:50:33,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28397.54 MB 2025-02-14 17:50:33,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28397.54 MB 2025-02-14 17:50:33,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:33,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20769.53 MB 2025-02-14 17:50:35,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:50:35,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:50:35,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.49 seconds 2025-02-14 17:50:35,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:35,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16844.20 MB 2025-02-14 17:50:35,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17258.26 MB 2025-02-14 17:50:35,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 414.06 MB 2025-02-14 17:50:35,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28397.54 MB 2025-02-14 17:50:35,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27453.82 MB 2025-02-14 17:50:35,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 17:50:35,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21183.72 MB 2025-02-14 17:50:35,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:50:35,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:50:35,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:50:35,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:35,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17258.26 MB 2025-02-14 17:50:35,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18732.55 MB 2025-02-14 17:50:35,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1474.30 MB 2025-02-14 17:50:35,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27453.82 MB 2025-02-14 17:50:35,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27453.82 MB 2025-02-14 17:50:35,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:35,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19838.15 MB 2025-02-14 17:50:35,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:50:35,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:50:35,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:50:35,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:35,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18732.55 MB 2025-02-14 17:50:35,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20481.22 MB 2025-02-14 17:50:35,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1748.66 MB 2025-02-14 17:50:35,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27453.82 MB 2025-02-14 17:50:35,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27453.82 MB 2025-02-14 17:50:35,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:35,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24805.74 MB 2025-02-14 17:50:35,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:50:35,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:50:35,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 17:50:35,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:35,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17258.26 MB 2025-02-14 17:50:35,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20481.22 MB 2025-02-14 17:50:35,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3222.96 MB 2025-02-14 17:50:35,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27453.82 MB 2025-02-14 17:50:35,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27453.82 MB 2025-02-14 17:50:35,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:35,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24805.74 MB 2025-02-14 17:50:35,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:50:35,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:50:35,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 17:50:35,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:35,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21677.38 MB 2025-02-14 17:50:35,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22275.64 MB 2025-02-14 17:50:35,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 598.26 MB 2025-02-14 17:50:35,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27453.82 MB 2025-02-14 17:50:35,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27778.88 MB 2025-02-14 17:50:35,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 325.06 MB 2025-02-14 17:50:35,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22827.72 MB 2025-02-14 17:50:35,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:50:35,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:50:35,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:50:35,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:35,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22597.70 MB 2025-02-14 17:50:35,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22827.03 MB 2025-02-14 17:50:35,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.34 MB 2025-02-14 17:50:35,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27778.88 MB 2025-02-14 17:50:35,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27778.88 MB 2025-02-14 17:50:35,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:35,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22960.88 MB 2025-02-14 17:50:35,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:50:35,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:50:35,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.88 seconds 2025-02-14 17:50:35,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:35,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14097.55 MB 2025-02-14 17:50:35,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23028.10 MB 2025-02-14 17:50:35,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8930.56 MB 2025-02-14 17:50:35,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57298.39 MB 2025-02-14 17:50:35,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27778.88 MB 2025-02-14 17:50:35,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29519.51 MB 2025-02-14 17:50:35,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23028.10 MB 2025-02-14 17:50:36,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:50:36,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:50:36,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:50:36,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:36,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23028.10 MB 2025-02-14 17:50:36,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26042.14 MB 2025-02-14 17:50:36,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 17:50:36,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27778.88 MB 2025-02-14 17:50:36,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27778.88 MB 2025-02-14 17:50:36,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:50:36,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26343.51 MB 2025-02-14 17:50:36,075 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 17:50:36,075 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1,'] 2025-02-14 17:50:36,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:50:36,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:50:36,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:50:36,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:50:36,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18686.64 MB 2025-02-14 17:50:36,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27125.66 MB 2025-02-14 17:50:36,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 17:50:36,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27778.88 MB 2025-02-14 17:50:36,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36169.58 MB 2025-02-14 17:50:36,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 17:50:36,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27125.66 MB 2025-02-14 17:50:36,239 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 17:50:36,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:36,240 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:50:36,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:36,241 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:50:36,246 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:50:36,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:50:36,247 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:50:36,247 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1,'] 2025-02-14 17:51:22,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:51:22,517 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:51:22,525 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:51:22,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:51:22,531 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:51:22,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:51:22,533 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:51:26,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:51:26,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:51:26,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.07 seconds 2025-02-14 17:51:26,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:26,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14759.52 MB 2025-02-14 17:51:26,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15669.03 MB 2025-02-14 17:51:26,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 909.51 MB 2025-02-14 17:51:26,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48754.59 MB 2025-02-14 17:51:26,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23276.29 MB 2025-02-14 17:51:26,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25478.30 MB 2025-02-14 17:51:26,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24683.88 MB 2025-02-14 17:51:26,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:51:26,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:51:26,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:51:26,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:26,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15669.03 MB 2025-02-14 17:51:26,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16109.62 MB 2025-02-14 17:51:26,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.59 MB 2025-02-14 17:51:26,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23276.29 MB 2025-02-14 17:51:26,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23276.29 MB 2025-02-14 17:51:26,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:51:26,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19282.40 MB 2025-02-14 17:51:27,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:51:27,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:51:27,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.23 seconds 2025-02-14 17:51:27,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:27,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16109.62 MB 2025-02-14 17:51:27,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16450.69 MB 2025-02-14 17:51:27,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.07 MB 2025-02-14 17:51:27,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23276.29 MB 2025-02-14 17:51:27,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23276.29 MB 2025-02-14 17:51:27,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:51:27,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20449.14 MB 2025-02-14 17:51:27,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:51:27,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:51:27,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:51:27,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:27,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16450.69 MB 2025-02-14 17:51:27,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17664.50 MB 2025-02-14 17:51:27,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1213.82 MB 2025-02-14 17:51:27,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23276.29 MB 2025-02-14 17:51:27,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23276.29 MB 2025-02-14 17:51:27,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:51:27,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18575.20 MB 2025-02-14 17:51:28,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:51:28,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:51:28,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 17:51:28,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:28,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17664.50 MB 2025-02-14 17:51:28,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19104.92 MB 2025-02-14 17:51:28,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1440.42 MB 2025-02-14 17:51:28,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23276.29 MB 2025-02-14 17:51:28,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24188.55 MB 2025-02-14 17:51:28,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 912.26 MB 2025-02-14 17:51:28,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22668.02 MB 2025-02-14 17:51:28,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:51:28,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:51:28,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 17:51:28,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:28,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16450.69 MB 2025-02-14 17:51:28,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19104.92 MB 2025-02-14 17:51:28,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2654.23 MB 2025-02-14 17:51:28,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23276.29 MB 2025-02-14 17:51:28,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24188.55 MB 2025-02-14 17:51:28,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 912.26 MB 2025-02-14 17:51:28,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22668.02 MB 2025-02-14 17:51:28,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:51:28,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:51:28,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 17:51:28,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:28,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20090.22 MB 2025-02-14 17:51:28,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20583.94 MB 2025-02-14 17:51:28,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 493.72 MB 2025-02-14 17:51:28,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24188.55 MB 2025-02-14 17:51:28,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24454.89 MB 2025-02-14 17:51:28,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 266.34 MB 2025-02-14 17:51:28,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21038.69 MB 2025-02-14 17:51:28,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:51:28,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:51:28,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:51:28,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:28,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20849.22 MB 2025-02-14 17:51:28,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21080.14 MB 2025-02-14 17:51:28,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.92 MB 2025-02-14 17:51:28,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24454.89 MB 2025-02-14 17:51:28,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24454.89 MB 2025-02-14 17:51:28,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:51:28,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21174.49 MB 2025-02-14 17:51:28,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:51:28,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:51:28,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.59 seconds 2025-02-14 17:51:28,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:28,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13864.11 MB 2025-02-14 17:51:28,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21281.22 MB 2025-02-14 17:51:28,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7417.10 MB 2025-02-14 17:51:28,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48754.59 MB 2025-02-14 17:51:28,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24454.89 MB 2025-02-14 17:51:28,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24299.70 MB 2025-02-14 17:51:28,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21281.22 MB 2025-02-14 17:51:28,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:51:28,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:51:28,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:51:28,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:28,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21281.22 MB 2025-02-14 17:51:28,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24295.25 MB 2025-02-14 17:51:28,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 17:51:28,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24454.89 MB 2025-02-14 17:51:28,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25662.85 MB 2025-02-14 17:51:28,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1207.96 MB 2025-02-14 17:51:28,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24596.88 MB 2025-02-14 17:51:28,413 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 17:51:28,413 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:51:28,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:51:28,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:51:28,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:51:28,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:51:28,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18194.56 MB 2025-02-14 17:51:28,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26633.59 MB 2025-02-14 17:51:28,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 17:51:28,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25662.85 MB 2025-02-14 17:51:28,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34053.55 MB 2025-02-14 17:51:28,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 17:51:28,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26633.59 MB 2025-02-14 17:51:28,580 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 17:51:28,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:51:28,582 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:51:28,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:51:28,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:51:28,587 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:51:28,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:51:28,588 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:51:28,589 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:52:17,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:52:17,195 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:52:17,200 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:52:17,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:52:17,205 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1015, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:52:17,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:52:17,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1015, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:52:32,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:52:32,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:52:32,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.56 seconds 2025-02-14 17:52:32,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:32,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20041.39 MB 2025-02-14 17:52:32,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23633.81 MB 2025-02-14 17:52:32,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3592.42 MB 2025-02-14 17:52:32,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46638.56 MB 2025-02-14 17:52:32,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29175.58 MB 2025-02-14 17:52:32,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17462.98 MB 2025-02-14 17:52:32,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32457.16 MB 2025-02-14 17:52:32,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:52:32,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:52:32,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 17:52:32,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:32,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23633.81 MB 2025-02-14 17:52:32,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21055.57 MB 2025-02-14 17:52:32,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2578.24 MB 2025-02-14 17:52:32,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29175.58 MB 2025-02-14 17:52:32,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39348.86 MB 2025-02-14 17:52:32,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10173.28 MB 2025-02-14 17:52:32,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34577.36 MB 2025-02-14 17:52:34,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:52:34,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:52:34,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 17:52:34,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:34,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21055.57 MB 2025-02-14 17:52:34,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21586.41 MB 2025-02-14 17:52:34,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:52:34,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39348.86 MB 2025-02-14 17:52:34,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 17:52:34,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11641.29 MB 2025-02-14 17:52:34,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25564.96 MB 2025-02-14 17:52:34,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:52:34,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:52:34,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:52:34,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:34,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21586.41 MB 2025-02-14 17:52:34,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23475.94 MB 2025-02-14 17:52:34,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:52:34,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 17:52:34,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 17:52:34,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:52:34,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24893.37 MB 2025-02-14 17:52:34,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:52:34,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:52:34,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:52:34,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:34,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23475.94 MB 2025-02-14 17:52:34,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25717.80 MB 2025-02-14 17:52:34,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:52:34,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 17:52:34,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33369.88 MB 2025-02-14 17:52:34,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 17:52:34,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31262.08 MB 2025-02-14 17:52:34,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:52:34,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:52:34,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:52:34,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:34,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21586.41 MB 2025-02-14 17:52:34,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25717.80 MB 2025-02-14 17:52:34,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:52:34,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 17:52:34,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33369.88 MB 2025-02-14 17:52:34,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 17:52:34,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31262.08 MB 2025-02-14 17:52:35,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:52:35,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:52:35,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:52:35,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:35,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27251.34 MB 2025-02-14 17:52:35,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28018.34 MB 2025-02-14 17:52:35,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:52:35,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33369.88 MB 2025-02-14 17:52:35,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 17:52:35,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:52:35,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28726.13 MB 2025-02-14 17:52:35,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:52:35,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:52:35,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:52:35,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:35,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28431.23 MB 2025-02-14 17:52:35,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28660.19 MB 2025-02-14 17:52:35,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 17:52:35,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33787.22 MB 2025-02-14 17:52:35,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 17:52:35,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:52:35,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28884.29 MB 2025-02-14 17:52:35,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:52:35,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:52:35,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.95 seconds 2025-02-14 17:52:35,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:35,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16505.05 MB 2025-02-14 17:52:35,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28861.07 MB 2025-02-14 17:52:35,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12356.02 MB 2025-02-14 17:52:35,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46638.56 MB 2025-02-14 17:52:35,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 17:52:35,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12851.35 MB 2025-02-14 17:52:35,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28884.29 MB 2025-02-14 17:52:35,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:52:35,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:52:35,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:52:35,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:35,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28861.07 MB 2025-02-14 17:52:35,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21506.39 MB 2025-02-14 17:52:35,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7354.68 MB 2025-02-14 17:52:35,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33787.22 MB 2025-02-14 17:52:35,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 17:52:35,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:52:35,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31370.28 MB 2025-02-14 17:52:35,442 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 17:52:35,442 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 17:52:35,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:52:35,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:52:35,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:52:35,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:52:35,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21506.39 MB 2025-02-14 17:52:35,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29937.06 MB 2025-02-14 17:52:35,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 17:52:35,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33787.22 MB 2025-02-14 17:52:35,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42169.53 MB 2025-02-14 17:52:35,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 17:52:35,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29937.06 MB 2025-02-14 17:52:35,609 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 17:52:35,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:52:35,610 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:52:35,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:52:35,611 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:52:35,616 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:52:35,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:52:35,617 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:52:35,617 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 17:53:34,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:53:34,669 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:53:34,674 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:53:34,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:53:34,678 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1147, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:53:34,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:53:34,679 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1147, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:53:52,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:53:52,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:53:52,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.56 seconds 2025-02-14 17:53:52,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:52,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20961.19 MB 2025-02-14 17:53:52,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25021.27 MB 2025-02-14 17:53:52,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4060.09 MB 2025-02-14 17:53:52,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54741.96 MB 2025-02-14 17:53:52,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30557.60 MB 2025-02-14 17:53:52,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24184.36 MB 2025-02-14 17:53:52,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33829.94 MB 2025-02-14 17:53:52,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:53:52,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:53:52,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 17:53:52,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:52,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25021.27 MB 2025-02-14 17:53:52,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21740.75 MB 2025-02-14 17:53:52,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3280.53 MB 2025-02-14 17:53:52,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30557.60 MB 2025-02-14 17:53:52,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42060.48 MB 2025-02-14 17:53:52,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11502.88 MB 2025-02-14 17:53:52,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36975.79 MB 2025-02-14 17:53:54,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:53:54,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:53:54,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 17:53:54,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21740.75 MB 2025-02-14 17:53:54,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22271.59 MB 2025-02-14 17:53:54,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:53:54,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42060.48 MB 2025-02-14 17:53:54,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27913.09 MB 2025-02-14 17:53:54,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14147.39 MB 2025-02-14 17:53:54,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26250.13 MB 2025-02-14 17:53:54,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:53:54,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:53:54,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:53:54,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22271.59 MB 2025-02-14 17:53:54,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24161.12 MB 2025-02-14 17:53:54,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:53:54,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27913.09 MB 2025-02-14 17:53:54,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27913.09 MB 2025-02-14 17:53:54,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:53:54,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25578.55 MB 2025-02-14 17:53:54,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:53:54,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:53:54,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:53:54,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24161.12 MB 2025-02-14 17:53:54,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26402.98 MB 2025-02-14 17:53:54,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:53:54,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27913.09 MB 2025-02-14 17:53:54,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34047.26 MB 2025-02-14 17:53:54,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 17:53:54,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31947.26 MB 2025-02-14 17:53:54,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:53:54,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:53:54,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:53:54,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22271.59 MB 2025-02-14 17:53:54,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26402.98 MB 2025-02-14 17:53:54,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:53:54,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27913.09 MB 2025-02-14 17:53:54,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34047.26 MB 2025-02-14 17:53:54,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 17:53:54,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31947.26 MB 2025-02-14 17:53:54,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:53:54,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:53:54,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:53:54,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27936.52 MB 2025-02-14 17:53:54,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28703.52 MB 2025-02-14 17:53:54,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:53:54,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34047.26 MB 2025-02-14 17:53:54,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 17:53:54,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:53:54,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29411.31 MB 2025-02-14 17:53:54,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:53:54,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:53:54,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:53:54,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29116.41 MB 2025-02-14 17:53:54,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29345.55 MB 2025-02-14 17:53:54,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.14 MB 2025-02-14 17:53:54,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-14 17:53:54,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 17:53:54,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:53:54,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29557.30 MB 2025-02-14 17:53:54,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:53:54,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:53:54,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.97 seconds 2025-02-14 17:53:54,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16964.95 MB 2025-02-14 17:53:54,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29545.44 MB 2025-02-14 17:53:54,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12580.49 MB 2025-02-14 17:53:54,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54741.96 MB 2025-02-14 17:53:54,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 17:53:54,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20277.36 MB 2025-02-14 17:53:54,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29557.30 MB 2025-02-14 17:53:54,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:53:54,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:53:54,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:53:54,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29545.44 MB 2025-02-14 17:53:54,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21951.64 MB 2025-02-14 17:53:54,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7593.80 MB 2025-02-14 17:53:54,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-14 17:53:54,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 17:53:54,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:53:54,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32042.36 MB 2025-02-14 17:53:54,932 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 17:53:54,932 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:53:54,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:53:54,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:53:54,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:53:54,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:53:54,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21951.64 MB 2025-02-14 17:53:54,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30340.78 MB 2025-02-14 17:53:54,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 17:53:54,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-14 17:53:54,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42807.07 MB 2025-02-14 17:53:54,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 17:53:54,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30340.78 MB 2025-02-14 17:53:55,094 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 17:53:55,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:53:55,096 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:53:55,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:53:55,097 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:53:55,102 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:53:55,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:53:55,103 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:53:55,103 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 17:54:28,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:28,669 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:54:28,674 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:54:28,677 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:28,677 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:54:28,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:28,678 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:54:48,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:54:48,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:54:48,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.51 seconds 2025-02-14 17:54:48,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:48,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-14 17:54:48,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-14 17:54:48,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-14 17:54:48,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51149.54 MB 2025-02-14 17:54:48,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-14 17:54:48,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16026.44 MB 2025-02-14 17:54:48,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.17 MB 2025-02-14 17:54:48,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:54:48,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:54:48,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 17:54:48,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:48,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-14 17:54:48,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-14 17:54:48,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-14 17:54:48,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35123.10 MB 2025-02-14 17:54:48,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44356.86 MB 2025-02-14 17:54:48,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9233.76 MB 2025-02-14 17:54:48,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39626.35 MB 2025-02-14 17:54:50,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:54:50,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:54:50,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 17:54:50,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-14 17:54:50,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-14 17:54:50,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:54:50,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44356.86 MB 2025-02-14 17:54:50,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30645.68 MB 2025-02-14 17:54:50,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13711.18 MB 2025-02-14 17:54:50,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26863.58 MB 2025-02-14 17:54:50,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:54:50,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:54:50,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:54:50,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 17:54:50,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-14 17:54:50,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:54:50,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30645.68 MB 2025-02-14 17:54:50,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30645.68 MB 2025-02-14 17:54:50,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:54:50,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-14 17:54:50,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:54:50,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:54:50,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:54:50,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-14 17:54:50,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 17:54:50,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:54:50,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30645.68 MB 2025-02-14 17:54:50,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34420.56 MB 2025-02-14 17:54:50,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 17:54:50,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 17:54:50,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:54:50,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:54:50,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:54:50,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 17:54:50,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 17:54:50,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:54:50,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30645.68 MB 2025-02-14 17:54:50,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34420.56 MB 2025-02-14 17:54:50,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 17:54:50,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 17:54:50,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:54:50,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:54:50,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 17:54:50,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-14 17:54:50,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-14 17:54:50,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:54:50,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34420.56 MB 2025-02-14 17:54:50,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34837.89 MB 2025-02-14 17:54:50,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:54:50,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-14 17:54:50,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:54:50,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:54:50,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:54:50,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-14 17:54:50,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29956.88 MB 2025-02-14 17:54:50,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.02 MB 2025-02-14 17:54:50,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34837.89 MB 2025-02-14 17:54:50,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34837.89 MB 2025-02-14 17:54:50,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:54:50,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30186.98 MB 2025-02-14 17:54:50,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:54:50,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:54:50,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.91 seconds 2025-02-14 17:54:50,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-14 17:54:50,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30157.36 MB 2025-02-14 17:54:50,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12781.29 MB 2025-02-14 17:54:50,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51149.54 MB 2025-02-14 17:54:50,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34837.89 MB 2025-02-14 17:54:50,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16311.65 MB 2025-02-14 17:54:50,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30186.98 MB 2025-02-14 17:54:50,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:54:50,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:54:50,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:54:50,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30157.36 MB 2025-02-14 17:54:50,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22363.83 MB 2025-02-14 17:54:50,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7793.53 MB 2025-02-14 17:54:50,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34837.89 MB 2025-02-14 17:54:50,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34837.89 MB 2025-02-14 17:54:50,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:54:50,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32655.20 MB 2025-02-14 17:54:50,876 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 17:54:50,877 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:54:50,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:54:50,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:54:50,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:54:50,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:54:50,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22363.83 MB 2025-02-14 17:54:50,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30756.42 MB 2025-02-14 17:54:50,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-14 17:54:50,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34837.89 MB 2025-02-14 17:54:50,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39011.22 MB 2025-02-14 17:54:50,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 17:54:50,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30756.42 MB 2025-02-14 17:54:51,038 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 17:54:51,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:51,039 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:54:51,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:51,040 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:54:51,045 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:54:51,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:51,046 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:54:51,046 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 17:54:57,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:57,390 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:54:57,398 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:54:57,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:57,404 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 811, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:54:57,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:54:57,406 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 811, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:55:10,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:55:10,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:55:10,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.63 seconds 2025-02-14 17:55:10,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:10,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18619.88 MB 2025-02-14 17:55:10,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21489.97 MB 2025-02-14 17:55:10,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2870.08 MB 2025-02-14 17:55:10,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51527.02 MB 2025-02-14 17:55:10,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26474.45 MB 2025-02-14 17:55:10,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25052.58 MB 2025-02-14 17:55:10,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30356.18 MB 2025-02-14 17:55:10,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:55:10,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:55:10,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 17:55:10,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:10,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21489.97 MB 2025-02-14 17:55:10,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19993.99 MB 2025-02-14 17:55:10,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1495.98 MB 2025-02-14 17:55:10,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26474.45 MB 2025-02-14 17:55:10,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33556.53 MB 2025-02-14 17:55:10,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7082.08 MB 2025-02-14 17:55:10,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30003.50 MB 2025-02-14 17:55:12,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:55:12,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:55:12,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 17:55:12,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19993.99 MB 2025-02-14 17:55:12,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20524.83 MB 2025-02-14 17:55:12,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:55:12,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33556.53 MB 2025-02-14 17:55:12,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27890.02 MB 2025-02-14 17:55:12,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5666.50 MB 2025-02-14 17:55:12,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24503.38 MB 2025-02-14 17:55:12,032 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:55:12,032 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:55:12,032 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:55:12,032 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,032 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20524.83 MB 2025-02-14 17:55:12,032 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22414.36 MB 2025-02-14 17:55:12,032 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:55:12,032 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 17:55:12,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27890.02 MB 2025-02-14 17:55:12,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:55:12,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23831.79 MB 2025-02-14 17:55:12,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:55:12,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:55:12,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 17:55:12,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22414.36 MB 2025-02-14 17:55:12,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24656.22 MB 2025-02-14 17:55:12,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:55:12,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 17:55:12,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31664.90 MB 2025-02-14 17:55:12,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 17:55:12,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30200.50 MB 2025-02-14 17:55:12,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:55:12,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:55:12,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:55:12,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20524.83 MB 2025-02-14 17:55:12,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24656.22 MB 2025-02-14 17:55:12,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:55:12,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 17:55:12,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31664.90 MB 2025-02-14 17:55:12,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 17:55:12,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30200.50 MB 2025-02-14 17:55:12,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:55:12,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:55:12,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 17:55:12,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26189.76 MB 2025-02-14 17:55:12,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26956.76 MB 2025-02-14 17:55:12,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:55:12,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31664.90 MB 2025-02-14 17:55:12,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32082.23 MB 2025-02-14 17:55:12,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:55:12,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27664.55 MB 2025-02-14 17:55:12,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:55:12,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:55:12,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:55:12,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27369.65 MB 2025-02-14 17:55:12,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27601.44 MB 2025-02-14 17:55:12,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.79 MB 2025-02-14 17:55:12,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32082.23 MB 2025-02-14 17:55:12,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32082.23 MB 2025-02-14 17:55:12,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:55:12,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27798.47 MB 2025-02-14 17:55:12,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:55:12,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:55:12,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.01 seconds 2025-02-14 17:55:12,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15794.30 MB 2025-02-14 17:55:12,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27802.51 MB 2025-02-14 17:55:12,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12008.22 MB 2025-02-14 17:55:12,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51527.02 MB 2025-02-14 17:55:12,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32082.23 MB 2025-02-14 17:55:12,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19444.79 MB 2025-02-14 17:55:12,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27802.51 MB 2025-02-14 17:55:12,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:55:12,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:55:12,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:55:12,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27802.51 MB 2025-02-14 17:55:12,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20798.68 MB 2025-02-14 17:55:12,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7003.83 MB 2025-02-14 17:55:12,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32082.23 MB 2025-02-14 17:55:12,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32082.23 MB 2025-02-14 17:55:12,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:55:12,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30314.18 MB 2025-02-14 17:55:12,705 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 17:55:12,706 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 1 ('] 2025-02-14 17:55:12,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:55:12,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:55:12,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:55:12,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:55:12,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20798.68 MB 2025-02-14 17:55:12,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29237.71 MB 2025-02-14 17:55:12,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 17:55:12,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32082.23 MB 2025-02-14 17:55:12,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40472.94 MB 2025-02-14 17:55:12,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 17:55:12,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29237.71 MB 2025-02-14 17:55:12,871 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 17:55:12,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:55:12,873 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:55:12,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:55:12,874 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:55:12,878 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:55:12,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:55:12,879 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:55:12,879 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 1 ('] 2025-02-14 17:57:12,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:57:12,928 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:57:12,936 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:57:12,943 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:57:12,943 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 106, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:57:12,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:57:12,945 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 106, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:57:14,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:57:14,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:57:14,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.69 seconds 2025-02-14 17:57:14,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:14,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-14 17:57:14,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14082.46 MB 2025-02-14 17:57:14,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 375.13 MB 2025-02-14 17:57:14,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53057.95 MB 2025-02-14 17:57:14,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 17:57:14,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33529.27 MB 2025-02-14 17:57:14,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22952.21 MB 2025-02-14 17:57:14,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:57:14,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:57:14,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 17:57:14,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:14,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14082.46 MB 2025-02-14 17:57:14,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14264.21 MB 2025-02-14 17:57:14,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 181.75 MB 2025-02-14 17:57:14,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 17:57:14,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 17:57:14,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:57:14,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14826.96 MB 2025-02-14 17:57:15,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:57:15,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:57:15,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.52 seconds 2025-02-14 17:57:15,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.21 MB 2025-02-14 17:57:15,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14404.88 MB 2025-02-14 17:57:15,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 140.67 MB 2025-02-14 17:57:15,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 17:57:15,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 17:57:15,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:57:15,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18348.92 MB 2025-02-14 17:57:15,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:57:15,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:57:15,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 17:57:15,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-14 17:57:15,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14905.42 MB 2025-02-14 17:57:15,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 500.60 MB 2025-02-14 17:57:15,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 17:57:15,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 17:57:15,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:57:15,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15281.05 MB 2025-02-14 17:57:15,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:57:15,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:57:15,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 17:57:15,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14905.42 MB 2025-02-14 17:57:15,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-14 17:57:15,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.03 MB 2025-02-14 17:57:15,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 17:57:15,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 17:57:15,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:57:15,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16968.75 MB 2025-02-14 17:57:15,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:57:15,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:57:15,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 17:57:15,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-14 17:57:15,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-14 17:57:15,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1108.64 MB 2025-02-14 17:57:15,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 17:57:15,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 17:57:15,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:57:15,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16968.75 MB 2025-02-14 17:57:15,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:57:15,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:57:15,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 17:57:15,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16101.38 MB 2025-02-14 17:57:15,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16356.73 MB 2025-02-14 17:57:15,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.36 MB 2025-02-14 17:57:15,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 17:57:15,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19692.26 MB 2025-02-14 17:57:15,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 17:57:15,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16544.30 MB 2025-02-14 17:57:15,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:57:15,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:57:15,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:57:15,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16518.26 MB 2025-02-14 17:57:15,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16746.54 MB 2025-02-14 17:57:15,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-14 17:57:15,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19692.26 MB 2025-02-14 17:57:15,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19692.26 MB 2025-02-14 17:57:15,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:57:15,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16746.54 MB 2025-02-14 17:57:15,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:57:15,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:57:15,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.47 seconds 2025-02-14 17:57:15,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13338.02 MB 2025-02-14 17:57:15,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16947.54 MB 2025-02-14 17:57:15,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3609.52 MB 2025-02-14 17:57:15,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53057.95 MB 2025-02-14 17:57:15,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19692.26 MB 2025-02-14 17:57:15,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33365.69 MB 2025-02-14 17:57:15,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16947.54 MB 2025-02-14 17:57:15,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:57:15,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:57:15,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 17:57:15,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14039.02 MB 2025-02-14 17:57:15,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17051.95 MB 2025-02-14 17:57:15,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.93 MB 2025-02-14 17:57:15,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19692.26 MB 2025-02-14 17:57:15,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19692.26 MB 2025-02-14 17:57:15,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:57:15,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17353.21 MB 2025-02-14 17:57:15,724 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 17:57:15,725 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 1 ('] 2025-02-14 17:57:15,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:57:15,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:57:15,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:57:15,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:57:15,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17051.95 MB 2025-02-14 17:57:15,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25487.55 MB 2025-02-14 17:57:15,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 17:57:15,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19692.26 MB 2025-02-14 17:57:15,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30178.02 MB 2025-02-14 17:57:15,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 17:57:15,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25487.55 MB 2025-02-14 17:57:15,979 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 17:57:15,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:57:15,982 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:57:15,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:57:15,984 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:57:15,991 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:57:15,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:57:15,994 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:57:15,994 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 1 ('] 2025-02-14 17:58:03,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:58:03,120 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 17:58:03,125 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 17:58:03,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:58:03,128 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2583, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 17:58:03,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:58:03,129 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2583, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 17:58:42,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 17:58:42,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 17:58:42,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.58 seconds 2025-02-14 17:58:42,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:42,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30968.17 MB 2025-02-14 17:58:42,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40109.65 MB 2025-02-14 17:58:42,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9141.49 MB 2025-02-14 17:58:42,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56568.58 MB 2025-02-14 17:58:42,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45256.54 MB 2025-02-14 17:58:42,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11312.04 MB 2025-02-14 17:58:42,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49250.75 MB 2025-02-14 17:58:42,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 17:58:42,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 17:58:42,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 17:58:42,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:42,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40109.65 MB 2025-02-14 17:58:42,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29206.77 MB 2025-02-14 17:58:42,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10902.89 MB 2025-02-14 17:58:42,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45256.54 MB 2025-02-14 17:58:42,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77030.49 MB 2025-02-14 17:58:42,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 31773.95 MB 2025-02-14 17:58:42,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66470.22 MB 2025-02-14 17:58:44,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 17:58:44,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 17:58:44,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 17:58:44,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:44,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29206.77 MB 2025-02-14 17:58:44,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29737.61 MB 2025-02-14 17:58:44,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 17:58:44,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77030.49 MB 2025-02-14 17:58:44,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32958.84 MB 2025-02-14 17:58:44,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -44071.65 MB 2025-02-14 17:58:44,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33717.19 MB 2025-02-14 17:58:44,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 17:58:44,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 17:58:44,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 17:58:44,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:44,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29737.61 MB 2025-02-14 17:58:44,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31627.14 MB 2025-02-14 17:58:44,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 17:58:44,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32958.84 MB 2025-02-14 17:58:44,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34846.28 MB 2025-02-14 17:58:44,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 17:58:44,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33044.57 MB 2025-02-14 17:58:45,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 17:58:45,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 17:58:45,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 17:58:45,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:45,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31627.14 MB 2025-02-14 17:58:45,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33869.00 MB 2025-02-14 17:58:45,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 17:58:45,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34846.28 MB 2025-02-14 17:58:45,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40980.45 MB 2025-02-14 17:58:45,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 17:58:45,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39413.28 MB 2025-02-14 17:58:45,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 17:58:45,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 17:58:45,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 17:58:45,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:45,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29737.61 MB 2025-02-14 17:58:45,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33869.00 MB 2025-02-14 17:58:45,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 17:58:45,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32958.84 MB 2025-02-14 17:58:45,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40980.45 MB 2025-02-14 17:58:45,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 17:58:45,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39413.28 MB 2025-02-14 17:58:45,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 17:58:45,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 17:58:45,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 17:58:45,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:45,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35402.54 MB 2025-02-14 17:58:45,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36169.54 MB 2025-02-14 17:58:45,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 17:58:45,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40980.45 MB 2025-02-14 17:58:45,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41397.78 MB 2025-02-14 17:58:45,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 17:58:45,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36877.33 MB 2025-02-14 17:58:45,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 17:58:45,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 17:58:45,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:58:45,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:45,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36582.43 MB 2025-02-14 17:58:45,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36809.82 MB 2025-02-14 17:58:45,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.39 MB 2025-02-14 17:58:45,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41397.78 MB 2025-02-14 17:58:45,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41397.78 MB 2025-02-14 17:58:45,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:58:45,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37032.75 MB 2025-02-14 17:58:45,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 17:58:45,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 17:58:45,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.11 seconds 2025-02-14 17:58:45,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:45,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21968.44 MB 2025-02-14 17:58:45,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37010.60 MB 2025-02-14 17:58:45,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15042.16 MB 2025-02-14 17:58:45,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47567.60 MB 2025-02-14 17:58:45,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41397.78 MB 2025-02-14 17:58:45,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6169.82 MB 2025-02-14 17:58:45,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37032.75 MB 2025-02-14 17:58:45,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 17:58:45,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 17:58:45,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 17:58:45,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:45,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37010.60 MB 2025-02-14 17:58:45,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26968.26 MB 2025-02-14 17:58:45,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10042.34 MB 2025-02-14 17:58:45,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41397.78 MB 2025-02-14 17:58:45,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41397.78 MB 2025-02-14 17:58:45,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 17:58:45,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39518.58 MB 2025-02-14 17:58:45,525 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 17:58:45,526 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 17:58:45,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 17:58:45,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 17:58:45,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 17:58:45,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 17:58:45,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26968.26 MB 2025-02-14 17:58:45,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35394.76 MB 2025-02-14 17:58:45,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 17:58:45,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41397.78 MB 2025-02-14 17:58:45,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45585.79 MB 2025-02-14 17:58:45,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 17:58:45,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35394.76 MB 2025-02-14 17:58:45,687 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 17:58:45,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:58:45,689 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 17:58:45,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:58:45,690 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 17:58:45,694 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 17:58:45,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 17:58:45,695 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 17:58:45,695 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 18:00:16,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:16,040 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:00:16,046 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:00:16,050 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:16,050 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1127, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:00:16,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:16,051 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1127, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:00:33,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:00:33,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:00:33,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.30 seconds 2025-02-14 18:00:33,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:33,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20821.82 MB 2025-02-14 18:00:33,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24810.61 MB 2025-02-14 18:00:33,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3988.78 MB 2025-02-14 18:00:33,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58151.93 MB 2025-02-14 18:00:33,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26289.90 MB 2025-02-14 18:00:33,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31862.03 MB 2025-02-14 18:00:33,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33691.39 MB 2025-02-14 18:00:33,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:00:33,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:00:33,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:00:33,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:33,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24810.61 MB 2025-02-14 18:00:33,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21637.82 MB 2025-02-14 18:00:33,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3172.79 MB 2025-02-14 18:00:33,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26289.90 MB 2025-02-14 18:00:33,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45583.70 MB 2025-02-14 18:00:33,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19293.80 MB 2025-02-14 18:00:33,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36786.54 MB 2025-02-14 18:00:35,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:00:35,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:00:35,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 18:00:35,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:35,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21637.82 MB 2025-02-14 18:00:35,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22168.66 MB 2025-02-14 18:00:35,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:00:35,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45583.70 MB 2025-02-14 18:00:35,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25134.37 MB 2025-02-14 18:00:35,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20449.33 MB 2025-02-14 18:00:35,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26149.05 MB 2025-02-14 18:00:35,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:00:35,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:00:35,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:00:35,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:35,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22168.66 MB 2025-02-14 18:00:35,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24058.20 MB 2025-02-14 18:00:35,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:00:35,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25134.37 MB 2025-02-14 18:00:35,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27021.80 MB 2025-02-14 18:00:35,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 18:00:35,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25475.62 MB 2025-02-14 18:00:35,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:00:35,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:00:35,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:00:35,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:35,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24058.20 MB 2025-02-14 18:00:35,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26300.05 MB 2025-02-14 18:00:35,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:00:35,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27021.80 MB 2025-02-14 18:00:35,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-14 18:00:35,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 18:00:35,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31844.33 MB 2025-02-14 18:00:35,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:00:35,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:00:35,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:00:35,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:35,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22168.66 MB 2025-02-14 18:00:35,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26300.05 MB 2025-02-14 18:00:35,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:00:35,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25134.37 MB 2025-02-14 18:00:35,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-14 18:00:35,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 18:00:35,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31844.33 MB 2025-02-14 18:00:35,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:00:35,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:00:35,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:00:35,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:35,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27833.59 MB 2025-02-14 18:00:35,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28600.60 MB 2025-02-14 18:00:35,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:00:35,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33627.83 MB 2025-02-14 18:00:35,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34045.17 MB 2025-02-14 18:00:35,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:00:35,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29308.38 MB 2025-02-14 18:00:35,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:00:35,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:00:35,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:00:35,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:35,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29013.49 MB 2025-02-14 18:00:35,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29241.56 MB 2025-02-14 18:00:35,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 18:00:35,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34045.17 MB 2025-02-14 18:00:35,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34045.17 MB 2025-02-14 18:00:35,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:00:35,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29460.07 MB 2025-02-14 18:00:35,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:00:35,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:00:35,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.72 seconds 2025-02-14 18:00:35,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:35,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16895.26 MB 2025-02-14 18:00:35,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29442.63 MB 2025-02-14 18:00:35,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12547.37 MB 2025-02-14 18:00:35,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58151.93 MB 2025-02-14 18:00:35,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34045.17 MB 2025-02-14 18:00:35,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24106.76 MB 2025-02-14 18:00:35,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29460.07 MB 2025-02-14 18:00:36,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:00:36,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:00:36,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:00:36,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:36,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29442.63 MB 2025-02-14 18:00:36,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21899.65 MB 2025-02-14 18:00:36,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7542.98 MB 2025-02-14 18:00:36,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34045.17 MB 2025-02-14 18:00:36,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34045.17 MB 2025-02-14 18:00:36,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:00:36,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31954.30 MB 2025-02-14 18:00:36,059 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:00:36,059 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:00:36,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:00:36,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:00:36,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:00:36,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:00:36,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21899.65 MB 2025-02-14 18:00:36,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30338.68 MB 2025-02-14 18:00:36,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:00:36,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34045.17 MB 2025-02-14 18:00:36,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42435.87 MB 2025-02-14 18:00:36,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:00:36,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30338.68 MB 2025-02-14 18:00:36,225 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:00:36,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:36,226 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:00:36,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:36,227 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:00:36,232 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:00:36,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:36,233 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:00:36,233 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:00:52,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:52,608 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:00:52,613 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:00:52,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:52,617 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2007, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:00:52,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:00:52,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2007, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:01:23,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:01:23,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:01:23,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.15 seconds 2025-02-14 18:01:23,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:23,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26953.80 MB 2025-02-14 18:01:23,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34056.86 MB 2025-02-14 18:01:23,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7103.05 MB 2025-02-14 18:01:23,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55020.88 MB 2025-02-14 18:01:23,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37792.78 MB 2025-02-14 18:01:23,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17228.10 MB 2025-02-14 18:01:23,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42994.26 MB 2025-02-14 18:01:23,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:01:23,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:01:23,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 18:01:23,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:23,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34056.86 MB 2025-02-14 18:01:23,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26212.67 MB 2025-02-14 18:01:23,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7844.19 MB 2025-02-14 18:01:23,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37792.78 MB 2025-02-14 18:01:23,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65064.14 MB 2025-02-14 18:01:23,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 27271.36 MB 2025-02-14 18:01:23,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54417.10 MB 2025-02-14 18:01:25,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:01:25,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:01:25,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 18:01:25,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:25,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26212.67 MB 2025-02-14 18:01:25,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26743.51 MB 2025-02-14 18:01:25,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:01:25,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65064.14 MB 2025-02-14 18:01:25,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32814.14 MB 2025-02-14 18:01:25,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32250.00 MB 2025-02-14 18:01:25,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30722.67 MB 2025-02-14 18:01:25,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:01:25,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:01:25,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:01:25,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:25,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26743.51 MB 2025-02-14 18:01:25,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28633.04 MB 2025-02-14 18:01:25,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:01:25,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32814.14 MB 2025-02-14 18:01:25,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32814.14 MB 2025-02-14 18:01:25,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:01:25,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30050.47 MB 2025-02-14 18:01:26,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:01:26,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:01:26,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:01:26,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:26,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28633.04 MB 2025-02-14 18:01:26,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30874.90 MB 2025-02-14 18:01:26,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:01:26,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32814.14 MB 2025-02-14 18:01:26,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38476.45 MB 2025-02-14 18:01:26,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:01:26,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36419.18 MB 2025-02-14 18:01:26,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:01:26,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:01:26,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:01:26,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:26,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26743.51 MB 2025-02-14 18:01:26,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30874.90 MB 2025-02-14 18:01:26,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:01:26,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32814.14 MB 2025-02-14 18:01:26,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38476.45 MB 2025-02-14 18:01:26,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:01:26,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36419.18 MB 2025-02-14 18:01:26,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:01:26,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:01:26,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 18:01:26,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:26,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32408.44 MB 2025-02-14 18:01:26,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33175.44 MB 2025-02-14 18:01:26,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:01:26,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38476.45 MB 2025-02-14 18:01:26,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 18:01:26,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:01:26,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33883.23 MB 2025-02-14 18:01:26,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:01:26,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:01:26,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:01:26,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:26,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33588.33 MB 2025-02-14 18:01:26,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33817.02 MB 2025-02-14 18:01:26,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-14 18:01:26,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38893.78 MB 2025-02-14 18:01:26,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 18:01:26,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:01:26,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34039.59 MB 2025-02-14 18:01:26,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:01:26,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:01:26,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.65 seconds 2025-02-14 18:01:26,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:26,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19961.25 MB 2025-02-14 18:01:26,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34017.87 MB 2025-02-14 18:01:26,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14056.62 MB 2025-02-14 18:01:26,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55020.88 MB 2025-02-14 18:01:26,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 18:01:26,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16127.10 MB 2025-02-14 18:01:26,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34039.59 MB 2025-02-14 18:01:26,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:01:26,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:01:26,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:01:26,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:26,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34017.87 MB 2025-02-14 18:01:26,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24962.22 MB 2025-02-14 18:01:26,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9055.66 MB 2025-02-14 18:01:26,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38893.78 MB 2025-02-14 18:01:26,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 18:01:26,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:01:26,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36526.78 MB 2025-02-14 18:01:26,553 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 18:01:26,554 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 18:01:26,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:01:26,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:01:26,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:01:26,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:01:26,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24962.22 MB 2025-02-14 18:01:26,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33392.61 MB 2025-02-14 18:01:26,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 18:01:26,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38893.78 MB 2025-02-14 18:01:26,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47274.00 MB 2025-02-14 18:01:26,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 18:01:26,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33392.61 MB 2025-02-14 18:01:26,716 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 18:01:26,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:01:26,717 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:01:26,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:01:26,718 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:01:26,723 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:01:26,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:01:26,724 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:01:26,724 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 18:01:57,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:01:57,942 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:01:57,947 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:01:57,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:01:57,951 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 331, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:01:57,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:01:57,952 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 331, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:02:03,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:02:03,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:02:03,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.12 seconds 2025-02-14 18:02:03,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15275.17 MB 2025-02-14 18:02:03,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16446.56 MB 2025-02-14 18:02:03,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.39 MB 2025-02-14 18:02:03,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55654.22 MB 2025-02-14 18:02:03,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28179.43 MB 2025-02-14 18:02:03,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27474.79 MB 2025-02-14 18:02:03,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25426.02 MB 2025-02-14 18:02:03,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:02:03,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:02:03,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:02:03,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16446.56 MB 2025-02-14 18:02:03,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.76 MB 2025-02-14 18:02:03,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1363.80 MB 2025-02-14 18:02:03,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28179.43 MB 2025-02-14 18:02:03,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28179.43 MB 2025-02-14 18:02:03,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:02:03,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17272.13 MB 2025-02-14 18:02:03,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:02:03,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:02:03,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:02:03,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.76 MB 2025-02-14 18:02:03,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15157.08 MB 2025-02-14 18:02:03,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 74.32 MB 2025-02-14 18:02:03,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28179.43 MB 2025-02-14 18:02:03,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27235.71 MB 2025-02-14 18:02:03,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 18:02:03,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18656.89 MB 2025-02-14 18:02:03,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:02:03,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:02:03,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:02:03,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15157.01 MB 2025-02-14 18:02:03,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15421.49 MB 2025-02-14 18:02:03,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.47 MB 2025-02-14 18:02:03,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27235.71 MB 2025-02-14 18:02:03,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27235.71 MB 2025-02-14 18:02:03,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:02:03,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15620.86 MB 2025-02-14 18:02:03,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:02:03,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:02:03,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 18:02:03,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15421.49 MB 2025-02-14 18:02:03,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15742.74 MB 2025-02-14 18:02:03,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 321.25 MB 2025-02-14 18:02:03,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27235.71 MB 2025-02-14 18:02:03,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27235.71 MB 2025-02-14 18:02:03,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:02:03,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16512.47 MB 2025-02-14 18:02:03,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:02:03,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:02:03,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:02:03,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15157.01 MB 2025-02-14 18:02:03,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15742.74 MB 2025-02-14 18:02:03,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 585.73 MB 2025-02-14 18:02:03,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27235.71 MB 2025-02-14 18:02:03,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27235.71 MB 2025-02-14 18:02:03,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:02:03,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16512.47 MB 2025-02-14 18:02:03,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:02:03,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:02:03,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 18:02:03,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16052.86 MB 2025-02-14 18:02:03,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16188.69 MB 2025-02-14 18:02:03,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 135.83 MB 2025-02-14 18:02:03,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27235.71 MB 2025-02-14 18:02:03,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27321.70 MB 2025-02-14 18:02:03,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 85.98 MB 2025-02-14 18:02:03,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16287.78 MB 2025-02-14 18:02:03,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:02:03,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:02:03,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:02:03,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16274.03 MB 2025-02-14 18:02:03,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16408.88 MB 2025-02-14 18:02:03,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.86 MB 2025-02-14 18:02:03,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27321.70 MB 2025-02-14 18:02:03,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27321.70 MB 2025-02-14 18:02:03,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:02:03,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16408.88 MB 2025-02-14 18:02:03,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:02:03,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:02:03,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.50 seconds 2025-02-14 18:02:03,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14121.94 MB 2025-02-14 18:02:03,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16530.40 MB 2025-02-14 18:02:03,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2408.46 MB 2025-02-14 18:02:03,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55654.22 MB 2025-02-14 18:02:03,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27321.70 MB 2025-02-14 18:02:03,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28332.52 MB 2025-02-14 18:02:03,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16530.40 MB 2025-02-14 18:02:03,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:02:03,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:02:03,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 18:02:03,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16530.40 MB 2025-02-14 18:02:03,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16270.90 MB 2025-02-14 18:02:03,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -259.50 MB 2025-02-14 18:02:03,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27321.70 MB 2025-02-14 18:02:03,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27321.70 MB 2025-02-14 18:02:03,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:02:03,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18412.56 MB 2025-02-14 18:02:03,623 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 4927, cut from 4929 2025-02-14 18:02:03,623 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:02:03,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:02:03,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:02:03,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:02:03,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:02:03,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16270.90 MB 2025-02-14 18:02:03,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21370.45 MB 2025-02-14 18:02:03,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5099.55 MB 2025-02-14 18:02:03,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27321.70 MB 2025-02-14 18:02:03,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27321.70 MB 2025-02-14 18:02:03,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:02:03,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21370.45 MB 2025-02-14 18:02:03,721 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 4719] 2025-02-14 18:02:03,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:02:03,723 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:02:03,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:02:03,724 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:02:03,728 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:02:03,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:02:03,729 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:02:03,729 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:03:49,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:03:49,039 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:03:49,044 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:03:49,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:03:49,047 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:03:49,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:03:49,048 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:03:58,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:03:58,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:03:58,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.91 seconds 2025-02-14 18:03:58,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:03:58,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17498.01 MB 2025-02-14 18:03:58,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19798.32 MB 2025-02-14 18:03:58,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2300.31 MB 2025-02-14 18:03:58,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32392.61 MB 2025-02-14 18:03:58,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23181.92 MB 2025-02-14 18:03:58,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9210.69 MB 2025-02-14 18:03:58,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28781.32 MB 2025-02-14 18:03:59,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:03:59,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:03:59,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 18:03:59,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:03:59,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19798.32 MB 2025-02-14 18:03:59,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19158.05 MB 2025-02-14 18:03:59,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -640.28 MB 2025-02-14 18:03:59,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23181.92 MB 2025-02-14 18:03:59,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32514.24 MB 2025-02-14 18:03:59,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9332.33 MB 2025-02-14 18:03:59,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28520.54 MB 2025-02-14 18:04:00,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:04:00,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:04:00,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:04:00,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:00,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19158.05 MB 2025-02-14 18:04:00,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19688.89 MB 2025-02-14 18:04:00,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:04:00,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32514.24 MB 2025-02-14 18:04:00,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25306.33 MB 2025-02-14 18:04:00,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7207.91 MB 2025-02-14 18:04:00,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23667.44 MB 2025-02-14 18:04:00,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:04:00,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:04:00,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:04:00,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:00,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19688.89 MB 2025-02-14 18:04:00,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21578.42 MB 2025-02-14 18:04:00,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:04:00,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25306.33 MB 2025-02-14 18:04:00,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25306.33 MB 2025-02-14 18:04:00,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:04:00,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22995.85 MB 2025-02-14 18:04:01,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:04:01,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:04:01,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:04:01,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:01,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21578.42 MB 2025-02-14 18:04:01,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23820.28 MB 2025-02-14 18:04:01,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:04:01,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25306.33 MB 2025-02-14 18:04:01,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31442.60 MB 2025-02-14 18:04:01,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 18:04:01,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29364.56 MB 2025-02-14 18:04:01,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:04:01,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:04:01,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:04:01,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:01,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19688.89 MB 2025-02-14 18:04:01,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23820.28 MB 2025-02-14 18:04:01,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:04:01,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25306.33 MB 2025-02-14 18:04:01,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31442.60 MB 2025-02-14 18:04:01,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 18:04:01,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29364.56 MB 2025-02-14 18:04:01,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:04:01,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:04:01,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:04:01,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:01,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25353.82 MB 2025-02-14 18:04:01,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26120.82 MB 2025-02-14 18:04:01,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:04:01,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31442.60 MB 2025-02-14 18:04:01,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31859.93 MB 2025-02-14 18:04:01,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:04:01,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26828.61 MB 2025-02-14 18:04:01,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:04:01,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:04:01,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:04:01,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:01,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26533.71 MB 2025-02-14 18:04:01,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26762.75 MB 2025-02-14 18:04:01,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.04 MB 2025-02-14 18:04:01,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31859.93 MB 2025-02-14 18:04:01,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31859.93 MB 2025-02-14 18:04:01,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:04:01,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26971.84 MB 2025-02-14 18:04:01,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:04:01,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:04:01,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.27 seconds 2025-02-14 18:04:01,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:01,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15233.36 MB 2025-02-14 18:04:01,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26963.82 MB 2025-02-14 18:04:01,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11730.46 MB 2025-02-14 18:04:01,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32392.61 MB 2025-02-14 18:04:01,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31859.93 MB 2025-02-14 18:04:01,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -532.68 MB 2025-02-14 18:04:01,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26971.84 MB 2025-02-14 18:04:01,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:04:01,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:04:01,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:04:01,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:01,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26963.82 MB 2025-02-14 18:04:01,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20237.75 MB 2025-02-14 18:04:01,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6726.07 MB 2025-02-14 18:04:01,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31859.93 MB 2025-02-14 18:04:01,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31859.93 MB 2025-02-14 18:04:01,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:04:01,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29475.49 MB 2025-02-14 18:04:01,601 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:04:01,601 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:04:01,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:04:01,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:04:01,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:04:01,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:01,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20237.75 MB 2025-02-14 18:04:01,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28676.77 MB 2025-02-14 18:04:01,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:04:01,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31859.93 MB 2025-02-14 18:04:01,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42349.89 MB 2025-02-14 18:04:01,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 18:04:01,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28676.77 MB 2025-02-14 18:04:01,764 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:04:01,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:01,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:04:01,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:01,766 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:04:01,771 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:04:01,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:01,772 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:04:01,772 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:04:11,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:11,691 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:04:11,696 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:04:11,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:11,699 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:04:11,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:11,700 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:04:44,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:04:44,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:04:44,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.19 seconds 2025-02-14 18:04:44,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:44,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27964.19 MB 2025-02-14 18:04:44,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35581.04 MB 2025-02-14 18:04:44,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7616.86 MB 2025-02-14 18:04:44,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54934.90 MB 2025-02-14 18:04:44,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38312.87 MB 2025-02-14 18:04:44,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16622.03 MB 2025-02-14 18:04:44,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44457.63 MB 2025-02-14 18:04:45,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:04:45,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:04:45,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 18:04:45,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:45,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35581.04 MB 2025-02-14 18:04:45,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26966.48 MB 2025-02-14 18:04:45,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8614.57 MB 2025-02-14 18:04:45,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38312.87 MB 2025-02-14 18:04:45,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67773.66 MB 2025-02-14 18:04:45,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29460.79 MB 2025-02-14 18:04:45,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55797.80 MB 2025-02-14 18:04:46,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:04:46,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:04:46,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 18:04:46,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:46,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26966.48 MB 2025-02-14 18:04:46,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27497.32 MB 2025-02-14 18:04:46,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:04:46,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67773.66 MB 2025-02-14 18:04:46,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33529.27 MB 2025-02-14 18:04:46,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34244.40 MB 2025-02-14 18:04:46,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31475.86 MB 2025-02-14 18:04:46,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:04:46,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:04:46,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:04:46,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:46,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27497.32 MB 2025-02-14 18:04:46,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29386.85 MB 2025-02-14 18:04:46,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:04:46,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33529.27 MB 2025-02-14 18:04:46,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33529.27 MB 2025-02-14 18:04:46,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:04:46,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30804.28 MB 2025-02-14 18:04:47,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:04:47,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:04:47,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:04:47,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:47,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29386.85 MB 2025-02-14 18:04:47,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31628.71 MB 2025-02-14 18:04:47,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:04:47,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33529.27 MB 2025-02-14 18:04:47,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39191.58 MB 2025-02-14 18:04:47,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:04:47,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37172.99 MB 2025-02-14 18:04:47,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:04:47,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:04:47,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:04:47,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:47,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27497.32 MB 2025-02-14 18:04:47,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31628.71 MB 2025-02-14 18:04:47,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:04:47,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33529.27 MB 2025-02-14 18:04:47,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39191.58 MB 2025-02-14 18:04:47,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:04:47,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37172.99 MB 2025-02-14 18:04:47,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:04:47,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:04:47,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:04:47,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:47,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33162.25 MB 2025-02-14 18:04:47,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33929.25 MB 2025-02-14 18:04:47,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:04:47,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39191.58 MB 2025-02-14 18:04:47,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39608.91 MB 2025-02-14 18:04:47,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:04:47,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34637.04 MB 2025-02-14 18:04:47,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:04:47,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:04:47,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:04:47,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:47,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34342.14 MB 2025-02-14 18:04:47,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34571.72 MB 2025-02-14 18:04:47,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.58 MB 2025-02-14 18:04:47,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39608.91 MB 2025-02-14 18:04:47,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39608.91 MB 2025-02-14 18:04:47,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:04:47,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34782.82 MB 2025-02-14 18:04:47,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:04:47,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:04:47,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.67 seconds 2025-02-14 18:04:47,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:47,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20466.45 MB 2025-02-14 18:04:47,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34772.60 MB 2025-02-14 18:04:47,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14306.15 MB 2025-02-14 18:04:47,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54934.90 MB 2025-02-14 18:04:47,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39608.91 MB 2025-02-14 18:04:47,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15325.99 MB 2025-02-14 18:04:47,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34782.82 MB 2025-02-14 18:04:47,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:04:47,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:04:47,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:04:47,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:47,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34772.60 MB 2025-02-14 18:04:47,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25467.79 MB 2025-02-14 18:04:47,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9304.81 MB 2025-02-14 18:04:47,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39608.91 MB 2025-02-14 18:04:47,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39608.91 MB 2025-02-14 18:04:47,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:04:47,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37281.81 MB 2025-02-14 18:04:47,659 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 18:04:47,659 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:04:47,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:04:47,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:04:47,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:04:47,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:04:47,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25467.79 MB 2025-02-14 18:04:47,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33898.46 MB 2025-02-14 18:04:47,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 18:04:47,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39608.91 MB 2025-02-14 18:04:47,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47991.23 MB 2025-02-14 18:04:47,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 18:04:47,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33898.46 MB 2025-02-14 18:04:47,883 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 18:04:47,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:47,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:04:47,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:47,887 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:04:47,894 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:04:47,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:04:47,896 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:04:47,896 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:05:51,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:05:51,001 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:05:51,006 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:05:51,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:05:51,010 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:05:51,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:05:51,011 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:05:53,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:05:53,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:05:53,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-14 18:05:53,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:53,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14250.85 MB 2025-02-14 18:05:53,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14902.01 MB 2025-02-14 18:05:53,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-14 18:05:53,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60563.65 MB 2025-02-14 18:05:53,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25885.15 MB 2025-02-14 18:05:53,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34678.51 MB 2025-02-14 18:05:53,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23722.22 MB 2025-02-14 18:05:53,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:05:53,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:05:53,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:05:53,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:53,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14902.01 MB 2025-02-14 18:05:53,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15091.09 MB 2025-02-14 18:05:53,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 189.07 MB 2025-02-14 18:05:53,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25885.15 MB 2025-02-14 18:05:53,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25885.15 MB 2025-02-14 18:05:53,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:05:53,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17262.03 MB 2025-02-14 18:05:54,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:05:54,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:05:54,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 18:05:54,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:54,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15091.09 MB 2025-02-14 18:05:54,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.39 MB 2025-02-14 18:05:54,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.30 MB 2025-02-14 18:05:54,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25885.15 MB 2025-02-14 18:05:54,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25885.15 MB 2025-02-14 18:05:54,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:05:54,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19260.74 MB 2025-02-14 18:05:54,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:05:54,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:05:54,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:05:54,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:54,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.32 MB 2025-02-14 18:05:54,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16095.29 MB 2025-02-14 18:05:54,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 783.97 MB 2025-02-14 18:05:54,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25885.15 MB 2025-02-14 18:05:54,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25885.15 MB 2025-02-14 18:05:54,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:05:54,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16683.53 MB 2025-02-14 18:05:54,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:05:54,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:05:54,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:05:54,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:54,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16095.29 MB 2025-02-14 18:05:54,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17025.70 MB 2025-02-14 18:05:54,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 930.41 MB 2025-02-14 18:05:54,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25885.15 MB 2025-02-14 18:05:54,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25885.15 MB 2025-02-14 18:05:54,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:05:54,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19326.54 MB 2025-02-14 18:05:54,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:05:54,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:05:54,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:05:54,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:54,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.32 MB 2025-02-14 18:05:54,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17025.70 MB 2025-02-14 18:05:54,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1714.37 MB 2025-02-14 18:05:54,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25885.15 MB 2025-02-14 18:05:54,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25885.15 MB 2025-02-14 18:05:54,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:05:54,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19326.54 MB 2025-02-14 18:05:54,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:05:54,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:05:54,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:05:54,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:54,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17662.12 MB 2025-02-14 18:05:54,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17980.42 MB 2025-02-14 18:05:54,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.31 MB 2025-02-14 18:05:54,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25885.15 MB 2025-02-14 18:05:54,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26055.02 MB 2025-02-14 18:05:54,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-14 18:05:54,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18282.77 MB 2025-02-14 18:05:54,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:05:54,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:05:54,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:05:54,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:54,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18151.78 MB 2025-02-14 18:05:54,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18370.93 MB 2025-02-14 18:05:54,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.15 MB 2025-02-14 18:05:54,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26055.02 MB 2025-02-14 18:05:54,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26055.02 MB 2025-02-14 18:05:54,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:05:54,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18389.63 MB 2025-02-14 18:05:54,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:05:54,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:05:54,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.90 seconds 2025-02-14 18:05:54,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:54,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13609.78 MB 2025-02-14 18:05:54,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18571.90 MB 2025-02-14 18:05:54,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4962.12 MB 2025-02-14 18:05:54,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60563.65 MB 2025-02-14 18:05:54,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26055.02 MB 2025-02-14 18:05:54,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34508.64 MB 2025-02-14 18:05:54,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18571.90 MB 2025-02-14 18:05:55,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:05:55,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:05:55,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 18:05:55,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:55,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18571.90 MB 2025-02-14 18:05:55,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17508.33 MB 2025-02-14 18:05:55,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1063.57 MB 2025-02-14 18:05:55,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26055.02 MB 2025-02-14 18:05:55,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26055.02 MB 2025-02-14 18:05:55,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:05:55,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19174.40 MB 2025-02-14 18:05:55,226 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 18:05:55,226 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:05:55,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:05:55,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:05:55,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:05:55,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:05:55,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17508.33 MB 2025-02-14 18:05:55,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25943.18 MB 2025-02-14 18:05:55,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 18:05:55,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26055.02 MB 2025-02-14 18:05:55,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34441.53 MB 2025-02-14 18:05:55,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 18:05:55,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25943.18 MB 2025-02-14 18:05:55,505 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 18:05:55,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:05:55,507 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:05:55,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:05:55,509 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:05:55,517 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:05:55,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:05:55,519 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:05:55,519 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:06:42,143 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:06:42,143 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:06:42,148 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:06:42,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:06:42,152 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1370, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:06:42,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:06:42,153 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1370, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:07:03,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:07:03,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:07:03,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.04 seconds 2025-02-14 18:07:03,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:03,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22515.09 MB 2025-02-14 18:07:03,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27363.70 MB 2025-02-14 18:07:03,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4848.62 MB 2025-02-14 18:07:03,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47020.24 MB 2025-02-14 18:07:03,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34219.23 MB 2025-02-14 18:07:03,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12801.02 MB 2025-02-14 18:07:03,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.81 MB 2025-02-14 18:07:03,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:07:03,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:07:03,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:07:03,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:03,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27363.70 MB 2025-02-14 18:07:03,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.05 MB 2025-02-14 18:07:03,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4463.65 MB 2025-02-14 18:07:03,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34219.23 MB 2025-02-14 18:07:03,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45944.41 MB 2025-02-14 18:07:03,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11725.18 MB 2025-02-14 18:07:03,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40121.76 MB 2025-02-14 18:07:05,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:07:05,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:07:05,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:07:05,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22900.05 MB 2025-02-14 18:07:05,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23430.89 MB 2025-02-14 18:07:05,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:07:05,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45944.41 MB 2025-02-14 18:07:05,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27919.38 MB 2025-02-14 18:07:05,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18025.02 MB 2025-02-14 18:07:05,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27410.48 MB 2025-02-14 18:07:05,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:07:05,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:07:05,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:07:05,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 18:07:05,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25320.43 MB 2025-02-14 18:07:05,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:07:05,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27919.38 MB 2025-02-14 18:07:05,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28863.10 MB 2025-02-14 18:07:05,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 18:07:05,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26737.86 MB 2025-02-14 18:07:05,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:07:05,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:07:05,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:07:05,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25320.43 MB 2025-02-14 18:07:05,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 18:07:05,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:07:05,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28863.10 MB 2025-02-14 18:07:05,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34997.27 MB 2025-02-14 18:07:05,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:07:05,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 18:07:05,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:07:05,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:07:05,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:07:05,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 18:07:05,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 18:07:05,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:07:05,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27919.38 MB 2025-02-14 18:07:05,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34997.27 MB 2025-02-14 18:07:05,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 18:07:05,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 18:07:05,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:07:05,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:07:05,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:07:05,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29095.83 MB 2025-02-14 18:07:05,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29862.83 MB 2025-02-14 18:07:05,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:07:05,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34997.27 MB 2025-02-14 18:07:05,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 18:07:05,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 18:07:05,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30570.62 MB 2025-02-14 18:07:05,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:07:05,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:07:05,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:07:05,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30275.72 MB 2025-02-14 18:07:05,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30504.16 MB 2025-02-14 18:07:05,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 18:07:05,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35410.41 MB 2025-02-14 18:07:05,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 18:07:05,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:07:05,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30720.52 MB 2025-02-14 18:07:05,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:07:05,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:07:05,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.44 seconds 2025-02-14 18:07:05,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17741.90 MB 2025-02-14 18:07:05,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30704.65 MB 2025-02-14 18:07:05,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12962.75 MB 2025-02-14 18:07:05,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47020.24 MB 2025-02-14 18:07:05,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 18:07:05,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11609.83 MB 2025-02-14 18:07:05,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30720.52 MB 2025-02-14 18:07:05,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:07:05,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:07:05,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:07:05,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30704.65 MB 2025-02-14 18:07:05,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22735.36 MB 2025-02-14 18:07:05,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7969.28 MB 2025-02-14 18:07:05,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35410.41 MB 2025-02-14 18:07:05,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 18:07:05,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:07:05,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33207.40 MB 2025-02-14 18:07:05,877 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 18:07:05,877 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:07:05,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:07:05,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:07:05,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:07:05,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:07:05,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22735.36 MB 2025-02-14 18:07:05,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31144.14 MB 2025-02-14 18:07:05,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8408.77 MB 2025-02-14 18:07:05,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35410.41 MB 2025-02-14 18:07:05,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39590.04 MB 2025-02-14 18:07:05,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 18:07:05,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31144.14 MB 2025-02-14 18:07:06,043 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 18:07:06,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:07:06,044 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:07:06,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:07:06,045 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:07:06,050 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:07:06,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:07:06,051 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:07:06,051 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:08:21,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:08:21,532 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:08:21,537 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:08:21,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:08:21,540 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1162, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:08:21,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:08:21,541 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1162, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:08:39,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:08:39,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:08:39,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.76 seconds 2025-02-14 18:08:39,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:39,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21065.71 MB 2025-02-14 18:08:39,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25178.22 MB 2025-02-14 18:08:39,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4112.52 MB 2025-02-14 18:08:39,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47949.28 MB 2025-02-14 18:08:39,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30586.96 MB 2025-02-14 18:08:39,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17362.32 MB 2025-02-14 18:08:39,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34160.96 MB 2025-02-14 18:08:39,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:08:39,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:08:39,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:08:39,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:39,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25178.22 MB 2025-02-14 18:08:39,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21818.73 MB 2025-02-14 18:08:39,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3359.50 MB 2025-02-14 18:08:39,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30586.96 MB 2025-02-14 18:08:39,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43257.95 MB 2025-02-14 18:08:39,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12670.99 MB 2025-02-14 18:08:39,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37599.42 MB 2025-02-14 18:08:41,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:08:41,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:08:41,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:08:41,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21818.73 MB 2025-02-14 18:08:41,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22349.57 MB 2025-02-14 18:08:41,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:08:41,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43257.95 MB 2025-02-14 18:08:41,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27890.02 MB 2025-02-14 18:08:41,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15367.93 MB 2025-02-14 18:08:41,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26328.12 MB 2025-02-14 18:08:41,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:08:41,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:08:41,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:08:41,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22349.57 MB 2025-02-14 18:08:41,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24239.10 MB 2025-02-14 18:08:41,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:08:41,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 18:08:41,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27890.02 MB 2025-02-14 18:08:41,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:08:41,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25656.53 MB 2025-02-14 18:08:41,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:08:41,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:08:41,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:08:41,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24239.10 MB 2025-02-14 18:08:41,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26480.96 MB 2025-02-14 18:08:41,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:08:41,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 18:08:41,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34024.19 MB 2025-02-14 18:08:41,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:08:41,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32025.24 MB 2025-02-14 18:08:41,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:08:41,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:08:41,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:08:41,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22349.57 MB 2025-02-14 18:08:41,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26480.96 MB 2025-02-14 18:08:41,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:08:41,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 18:08:41,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34024.19 MB 2025-02-14 18:08:41,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:08:41,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32025.24 MB 2025-02-14 18:08:41,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:08:41,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:08:41,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:08:41,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28014.50 MB 2025-02-14 18:08:41,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28781.50 MB 2025-02-14 18:08:41,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:08:41,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34024.19 MB 2025-02-14 18:08:41,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34439.43 MB 2025-02-14 18:08:41,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:08:41,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29489.29 MB 2025-02-14 18:08:41,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:08:41,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:08:41,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:08:41,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29194.39 MB 2025-02-14 18:08:41,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29423.43 MB 2025-02-14 18:08:41,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-14 18:08:41,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34439.43 MB 2025-02-14 18:08:41,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34439.43 MB 2025-02-14 18:08:41,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:08:41,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29649.48 MB 2025-02-14 18:08:41,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:08:41,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:08:41,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.15 seconds 2025-02-14 18:08:41,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17017.21 MB 2025-02-14 18:08:41,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29624.38 MB 2025-02-14 18:08:41,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12607.17 MB 2025-02-14 18:08:41,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47949.28 MB 2025-02-14 18:08:41,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34439.43 MB 2025-02-14 18:08:41,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13509.85 MB 2025-02-14 18:08:41,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29649.48 MB 2025-02-14 18:08:41,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:08:41,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:08:41,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:08:41,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29624.38 MB 2025-02-14 18:08:41,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22019.69 MB 2025-02-14 18:08:41,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7604.68 MB 2025-02-14 18:08:41,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34439.43 MB 2025-02-14 18:08:41,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34439.43 MB 2025-02-14 18:08:41,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:08:41,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.51 MB 2025-02-14 18:08:41,985 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 18:08:41,985 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:08:41,991 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:08:41,991 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:08:41,991 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:08:41,991 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:08:41,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22019.69 MB 2025-02-14 18:08:41,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30454.31 MB 2025-02-14 18:08:41,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 18:08:41,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34439.43 MB 2025-02-14 18:08:41,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42823.84 MB 2025-02-14 18:08:41,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 18:08:41,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30454.31 MB 2025-02-14 18:08:42,147 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 18:08:42,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:08:42,149 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:08:42,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:08:42,150 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:08:42,154 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:08:42,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:08:42,155 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:08:42,155 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:09:08,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:08,660 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:09:08,665 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:09:08,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:08,669 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1592, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:09:08,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:08,670 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1592, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:09:33,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:09:33,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:09:33,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.59 seconds 2025-02-14 18:09:33,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:33,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24062.02 MB 2025-02-14 18:09:33,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29697.06 MB 2025-02-14 18:09:33,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5635.05 MB 2025-02-14 18:09:33,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51208.26 MB 2025-02-14 18:09:33,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36299.60 MB 2025-02-14 18:09:33,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14908.65 MB 2025-02-14 18:09:33,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38517.03 MB 2025-02-14 18:09:33,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:09:33,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:09:33,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:09:33,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:33,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29697.06 MB 2025-02-14 18:09:33,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24054.16 MB 2025-02-14 18:09:33,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5642.90 MB 2025-02-14 18:09:33,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36299.60 MB 2025-02-14 18:09:33,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52718.21 MB 2025-02-14 18:09:33,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16418.60 MB 2025-02-14 18:09:33,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45784.20 MB 2025-02-14 18:09:35,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:09:35,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:09:35,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 18:09:35,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24054.16 MB 2025-02-14 18:09:35,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24585.00 MB 2025-02-14 18:09:35,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:09:35,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52718.21 MB 2025-02-14 18:09:35,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 18:09:35,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20638.07 MB 2025-02-14 18:09:35,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28563.55 MB 2025-02-14 18:09:35,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:09:35,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:09:35,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:09:35,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24585.00 MB 2025-02-14 18:09:35,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26474.54 MB 2025-02-14 18:09:35,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:09:35,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 18:09:35,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 18:09:35,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:09:35,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27891.97 MB 2025-02-14 18:09:35,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:09:35,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:09:35,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:09:35,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26474.54 MB 2025-02-14 18:09:35,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28716.39 MB 2025-02-14 18:09:35,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:09:35,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 18:09:35,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36798.73 MB 2025-02-14 18:09:35,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 18:09:35,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.68 MB 2025-02-14 18:09:35,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:09:35,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:09:35,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:09:35,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24585.00 MB 2025-02-14 18:09:35,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28716.39 MB 2025-02-14 18:09:35,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:09:35,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 18:09:35,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36798.73 MB 2025-02-14 18:09:35,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 18:09:35,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.68 MB 2025-02-14 18:09:35,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:09:35,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:09:35,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:09:35,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30249.94 MB 2025-02-14 18:09:35,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31016.94 MB 2025-02-14 18:09:35,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:09:35,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36798.73 MB 2025-02-14 18:09:35,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 18:09:35,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:09:35,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31724.73 MB 2025-02-14 18:09:35,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:09:35,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:09:35,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:09:35,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31429.83 MB 2025-02-14 18:09:35,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31657.88 MB 2025-02-14 18:09:35,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-14 18:09:35,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37216.06 MB 2025-02-14 18:09:35,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 18:09:35,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:09:35,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31866.06 MB 2025-02-14 18:09:35,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:09:35,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:09:35,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.03 seconds 2025-02-14 18:09:35,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18515.36 MB 2025-02-14 18:09:35,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31858.36 MB 2025-02-14 18:09:35,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13343.00 MB 2025-02-14 18:09:35,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51208.26 MB 2025-02-14 18:09:35,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 18:09:35,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13992.20 MB 2025-02-14 18:09:35,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31866.06 MB 2025-02-14 18:09:35,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:09:35,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:09:35,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:09:35,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31858.36 MB 2025-02-14 18:09:35,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23503.13 MB 2025-02-14 18:09:35,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8355.24 MB 2025-02-14 18:09:35,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37216.06 MB 2025-02-14 18:09:35,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 18:09:35,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:09:35,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34356.20 MB 2025-02-14 18:09:35,986 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 18:09:35,986 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:09:35,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:09:35,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:09:35,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:09:35,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:35,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23503.13 MB 2025-02-14 18:09:35,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31895.72 MB 2025-02-14 18:09:35,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-14 18:09:35,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37216.06 MB 2025-02-14 18:09:35,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45560.63 MB 2025-02-14 18:09:35,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-14 18:09:35,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31895.72 MB 2025-02-14 18:09:36,148 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 18:09:36,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:36,149 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:09:36,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:36,150 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:09:36,155 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:09:36,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:36,156 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:09:36,156 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:09:44,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:44,982 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:09:44,987 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:09:44,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:44,990 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 534, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:09:44,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:44,991 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 534, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:09:53,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:09:53,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:09:53,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.31 seconds 2025-02-14 18:09:53,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:53,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16689.70 MB 2025-02-14 18:09:53,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18579.50 MB 2025-02-14 18:09:53,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.80 MB 2025-02-14 18:09:53,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58076.43 MB 2025-02-14 18:09:53,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25597.84 MB 2025-02-14 18:09:53,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32478.59 MB 2025-02-14 18:09:53,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27520.03 MB 2025-02-14 18:09:53,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:09:53,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:09:53,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 18:09:53,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:53,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18579.50 MB 2025-02-14 18:09:53,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18555.00 MB 2025-02-14 18:09:53,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -24.50 MB 2025-02-14 18:09:53,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25597.84 MB 2025-02-14 18:09:53,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29490.15 MB 2025-02-14 18:09:53,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3892.31 MB 2025-02-14 18:09:53,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26331.06 MB 2025-02-14 18:09:55,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:09:55,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:09:55,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 18:09:55,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18555.00 MB 2025-02-14 18:09:55,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19085.84 MB 2025-02-14 18:09:55,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:09:55,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29490.15 MB 2025-02-14 18:09:55,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26306.67 MB 2025-02-14 18:09:55,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3183.48 MB 2025-02-14 18:09:55,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23064.39 MB 2025-02-14 18:09:55,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:09:55,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:09:55,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:09:55,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19085.84 MB 2025-02-14 18:09:55,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20975.37 MB 2025-02-14 18:09:55,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:09:55,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26306.67 MB 2025-02-14 18:09:55,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26306.67 MB 2025-02-14 18:09:55,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:09:55,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22392.80 MB 2025-02-14 18:09:55,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:09:55,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:09:55,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:09:55,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20975.37 MB 2025-02-14 18:09:55,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23217.23 MB 2025-02-14 18:09:55,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:09:55,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26306.67 MB 2025-02-14 18:09:55,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31025.27 MB 2025-02-14 18:09:55,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 18:09:55,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28761.51 MB 2025-02-14 18:09:55,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:09:55,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:09:55,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:09:55,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19085.84 MB 2025-02-14 18:09:55,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23217.23 MB 2025-02-14 18:09:55,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:09:55,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26306.67 MB 2025-02-14 18:09:55,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31025.27 MB 2025-02-14 18:09:55,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 18:09:55,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28761.51 MB 2025-02-14 18:09:55,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:09:55,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:09:55,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 18:09:55,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24750.77 MB 2025-02-14 18:09:55,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25517.77 MB 2025-02-14 18:09:55,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:09:55,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31025.27 MB 2025-02-14 18:09:55,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31440.50 MB 2025-02-14 18:09:55,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:09:55,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26225.56 MB 2025-02-14 18:09:55,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:09:55,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:09:55,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:09:55,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25930.66 MB 2025-02-14 18:09:55,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26158.43 MB 2025-02-14 18:09:55,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.77 MB 2025-02-14 18:09:55,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31440.50 MB 2025-02-14 18:09:55,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31440.50 MB 2025-02-14 18:09:55,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:09:55,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26354.24 MB 2025-02-14 18:09:55,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:09:55,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:09:55,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.71 seconds 2025-02-14 18:09:55,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14829.20 MB 2025-02-14 18:09:55,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26358.92 MB 2025-02-14 18:09:55,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11529.71 MB 2025-02-14 18:09:55,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58076.43 MB 2025-02-14 18:09:55,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31440.50 MB 2025-02-14 18:09:55,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26635.93 MB 2025-02-14 18:09:55,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26358.92 MB 2025-02-14 18:09:55,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:09:55,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:09:55,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:09:55,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26358.92 MB 2025-02-14 18:09:55,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19823.38 MB 2025-02-14 18:09:55,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6535.53 MB 2025-02-14 18:09:55,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31440.50 MB 2025-02-14 18:09:55,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31440.50 MB 2025-02-14 18:09:55,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:09:55,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28862.29 MB 2025-02-14 18:09:55,986 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 18:09:55,987 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:09:55,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:09:55,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:09:55,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:09:55,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:09:55,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19823.38 MB 2025-02-14 18:09:55,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28234.20 MB 2025-02-14 18:09:55,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 18:09:55,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31440.50 MB 2025-02-14 18:09:55,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39803.94 MB 2025-02-14 18:09:55,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 18:09:55,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28234.20 MB 2025-02-14 18:09:56,151 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 18:09:56,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:56,153 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:09:56,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:56,154 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:09:56,158 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:09:56,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:09:56,159 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:09:56,160 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:11:24,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:11:24,007 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:11:24,012 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:11:24,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:11:24,016 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:11:24,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:11:24,017 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:11:26,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:11:26,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:11:26,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.24 seconds 2025-02-14 18:11:26,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-14 18:11:26,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14492.24 MB 2025-02-14 18:11:26,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.15 MB 2025-02-14 18:11:26,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48167.39 MB 2025-02-14 18:11:26,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20711.47 MB 2025-02-14 18:11:26,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27455.91 MB 2025-02-14 18:11:26,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23450.46 MB 2025-02-14 18:11:26,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:11:26,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:11:26,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:11:26,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14492.24 MB 2025-02-14 18:11:26,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14150.92 MB 2025-02-14 18:11:26,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -341.32 MB 2025-02-14 18:11:26,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20711.47 MB 2025-02-14 18:11:26,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20711.47 MB 2025-02-14 18:11:26,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:11:26,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15352.65 MB 2025-02-14 18:11:26,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:11:26,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:11:26,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 18:11:26,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14150.92 MB 2025-02-14 18:11:26,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14231.87 MB 2025-02-14 18:11:26,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-14 18:11:26,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20711.47 MB 2025-02-14 18:11:26,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20711.47 MB 2025-02-14 18:11:26,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:11:26,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18044.17 MB 2025-02-14 18:11:26,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:11:26,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:11:26,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:11:26,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14231.81 MB 2025-02-14 18:11:26,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14519.89 MB 2025-02-14 18:11:26,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-14 18:11:26,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20711.47 MB 2025-02-14 18:11:26,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20711.47 MB 2025-02-14 18:11:26,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:11:26,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14736.06 MB 2025-02-14 18:11:26,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:11:26,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:11:26,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:11:26,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14519.89 MB 2025-02-14 18:11:26,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.32 MB 2025-02-14 18:11:26,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 350.42 MB 2025-02-14 18:11:26,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20711.47 MB 2025-02-14 18:11:26,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20711.47 MB 2025-02-14 18:11:26,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:11:26,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15707.77 MB 2025-02-14 18:11:26,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:11:26,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:11:26,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:11:26,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14231.81 MB 2025-02-14 18:11:26,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.32 MB 2025-02-14 18:11:26,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 638.51 MB 2025-02-14 18:11:26,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20711.47 MB 2025-02-14 18:11:26,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20711.47 MB 2025-02-14 18:11:26,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:11:26,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15707.77 MB 2025-02-14 18:11:26,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:11:26,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:11:26,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 18:11:26,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15209.03 MB 2025-02-14 18:11:26,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15355.98 MB 2025-02-14 18:11:26,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.95 MB 2025-02-14 18:11:26,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20711.47 MB 2025-02-14 18:11:26,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20801.65 MB 2025-02-14 18:11:26,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 90.18 MB 2025-02-14 18:11:26,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15463.92 MB 2025-02-14 18:11:26,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:11:26,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:11:26,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:11:26,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15448.94 MB 2025-02-14 18:11:26,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15596.19 MB 2025-02-14 18:11:26,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.25 MB 2025-02-14 18:11:26,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20801.65 MB 2025-02-14 18:11:26,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20801.65 MB 2025-02-14 18:11:26,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:11:26,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15596.19 MB 2025-02-14 18:11:26,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:11:26,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:11:26,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.64 seconds 2025-02-14 18:11:26,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13473.90 MB 2025-02-14 18:11:26,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15728.03 MB 2025-02-14 18:11:26,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2254.13 MB 2025-02-14 18:11:26,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48167.39 MB 2025-02-14 18:11:26,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20801.65 MB 2025-02-14 18:11:26,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27365.74 MB 2025-02-14 18:11:26,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15728.03 MB 2025-02-14 18:11:26,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:11:26,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:11:26,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 18:11:26,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15728.03 MB 2025-02-14 18:11:26,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15806.07 MB 2025-02-14 18:11:26,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 78.03 MB 2025-02-14 18:11:26,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20801.65 MB 2025-02-14 18:11:26,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20801.65 MB 2025-02-14 18:11:26,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:11:26,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17638.43 MB 2025-02-14 18:11:26,841 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-14 18:11:26,841 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 18:11:26,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:11:26,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:11:26,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:11:26,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:11:26,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15806.07 MB 2025-02-14 18:11:26,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21339.13 MB 2025-02-14 18:11:26,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.07 MB 2025-02-14 18:11:26,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20801.65 MB 2025-02-14 18:11:26,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23553.11 MB 2025-02-14 18:11:26,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2751.46 MB 2025-02-14 18:11:26,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21339.13 MB 2025-02-14 18:11:26,948 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-14 18:11:26,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:11:26,949 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:11:26,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:11:26,950 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:11:26,955 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:11:26,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:11:26,956 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:11:26,956 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 18:12:16,737 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:12:16,737 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:12:16,742 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:12:16,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:12:16,746 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1956, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:12:16,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:12:16,747 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1956, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:12:46,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:12:46,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:12:46,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.07 seconds 2025-02-14 18:12:46,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:46,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26598.43 MB 2025-02-14 18:12:46,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33521.13 MB 2025-02-14 18:12:46,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6922.70 MB 2025-02-14 18:12:46,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39229.33 MB 2025-02-14 18:12:46,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35211.18 MB 2025-02-14 18:12:46,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4018.14 MB 2025-02-14 18:12:46,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42412.39 MB 2025-02-14 18:12:46,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:12:46,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:12:46,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 18:12:46,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:46,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33521.13 MB 2025-02-14 18:12:46,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25947.53 MB 2025-02-14 18:12:46,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7573.59 MB 2025-02-14 18:12:46,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35211.18 MB 2025-02-14 18:12:46,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65301.12 MB 2025-02-14 18:12:46,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30089.94 MB 2025-02-14 18:12:46,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53356.52 MB 2025-02-14 18:12:48,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:12:48,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:12:48,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 18:12:48,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:48,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25947.53 MB 2025-02-14 18:12:48,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26478.37 MB 2025-02-14 18:12:48,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:12:48,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65301.12 MB 2025-02-14 18:12:48,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27659.34 MB 2025-02-14 18:12:48,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37641.78 MB 2025-02-14 18:12:48,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30457.96 MB 2025-02-14 18:12:48,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:12:48,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:12:48,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:12:48,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:48,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26478.37 MB 2025-02-14 18:12:48,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28367.48 MB 2025-02-14 18:12:48,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.11 MB 2025-02-14 18:12:48,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27659.34 MB 2025-02-14 18:12:48,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30962.35 MB 2025-02-14 18:12:48,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 18:12:48,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29784.91 MB 2025-02-14 18:12:49,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:12:49,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:12:49,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:12:49,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:49,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28367.48 MB 2025-02-14 18:12:49,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30609.34 MB 2025-02-14 18:12:49,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:12:49,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30962.35 MB 2025-02-14 18:12:49,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37568.38 MB 2025-02-14 18:12:49,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 18:12:49,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36153.62 MB 2025-02-14 18:12:49,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:12:49,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:12:49,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:12:49,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:49,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26478.37 MB 2025-02-14 18:12:49,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30609.34 MB 2025-02-14 18:12:49,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.96 MB 2025-02-14 18:12:49,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27659.34 MB 2025-02-14 18:12:49,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37568.38 MB 2025-02-14 18:12:49,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 18:12:49,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36153.62 MB 2025-02-14 18:12:49,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:12:49,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:12:49,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:12:49,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:49,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32142.88 MB 2025-02-14 18:12:49,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32909.88 MB 2025-02-14 18:12:49,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:12:49,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37568.38 MB 2025-02-14 18:12:49,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37983.62 MB 2025-02-14 18:12:49,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:12:49,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33617.67 MB 2025-02-14 18:12:49,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:12:49,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:12:49,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:12:49,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:49,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33322.77 MB 2025-02-14 18:12:49,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33551.23 MB 2025-02-14 18:12:49,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-14 18:12:49,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37983.62 MB 2025-02-14 18:12:49,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37983.62 MB 2025-02-14 18:12:49,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:12:49,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33768.24 MB 2025-02-14 18:12:49,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:12:49,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:12:49,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.57 seconds 2025-02-14 18:12:49,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:49,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19783.57 MB 2025-02-14 18:12:49,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33752.03 MB 2025-02-14 18:12:49,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13968.47 MB 2025-02-14 18:12:49,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32411.48 MB 2025-02-14 18:12:49,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37983.62 MB 2025-02-14 18:12:49,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5572.13 MB 2025-02-14 18:12:49,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33768.24 MB 2025-02-14 18:12:49,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:12:49,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:12:49,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:12:49,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:49,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33752.03 MB 2025-02-14 18:12:49,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24783.34 MB 2025-02-14 18:12:49,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8968.69 MB 2025-02-14 18:12:49,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37983.62 MB 2025-02-14 18:12:49,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37983.62 MB 2025-02-14 18:12:49,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:12:49,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36260.32 MB 2025-02-14 18:12:49,603 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 18:12:49,603 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 18:12:49,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:12:49,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:12:49,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:12:49,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:12:49,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24783.34 MB 2025-02-14 18:12:49,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33210.67 MB 2025-02-14 18:12:49,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 18:12:49,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37983.62 MB 2025-02-14 18:12:49,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46363.84 MB 2025-02-14 18:12:49,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 18:12:49,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33210.67 MB 2025-02-14 18:12:49,769 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 18:12:49,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:12:49,770 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:12:49,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:12:49,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:12:49,776 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:12:49,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:12:49,777 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:12:49,777 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 18:14:13,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:14:13,608 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:14:13,613 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:14:13,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:14:13,617 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1248, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:14:13,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:14:13,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1248, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:14:32,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:14:32,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:14:32,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.06 seconds 2025-02-14 18:14:32,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:32,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21664.97 MB 2025-02-14 18:14:32,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26081.57 MB 2025-02-14 18:14:32,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4416.60 MB 2025-02-14 18:14:32,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54744.06 MB 2025-02-14 18:14:32,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33432.80 MB 2025-02-14 18:14:32,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21311.26 MB 2025-02-14 18:14:32,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34986.71 MB 2025-02-14 18:14:32,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:14:32,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:14:32,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:14:32,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:32,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26081.57 MB 2025-02-14 18:14:32,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22265.81 MB 2025-02-14 18:14:32,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3815.76 MB 2025-02-14 18:14:32,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33432.80 MB 2025-02-14 18:14:32,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45197.82 MB 2025-02-14 18:14:32,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11765.02 MB 2025-02-14 18:14:32,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39203.98 MB 2025-02-14 18:14:34,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:14:34,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:14:34,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:14:34,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:34,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22265.81 MB 2025-02-14 18:14:34,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22796.65 MB 2025-02-14 18:14:34,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:14:34,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45197.82 MB 2025-02-14 18:14:34,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27076.33 MB 2025-02-14 18:14:34,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18121.49 MB 2025-02-14 18:14:34,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26775.20 MB 2025-02-14 18:14:34,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:14:34,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:14:34,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:14:34,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:34,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.65 MB 2025-02-14 18:14:34,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24686.19 MB 2025-02-14 18:14:34,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:14:34,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27076.33 MB 2025-02-14 18:14:34,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28963.77 MB 2025-02-14 18:14:34,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 18:14:34,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26103.62 MB 2025-02-14 18:14:35,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:14:35,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:14:35,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:14:35,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:35,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24686.19 MB 2025-02-14 18:14:35,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26928.04 MB 2025-02-14 18:14:35,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:14:35,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28963.77 MB 2025-02-14 18:14:35,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34626.08 MB 2025-02-14 18:14:35,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:14:35,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32472.33 MB 2025-02-14 18:14:35,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:14:35,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:14:35,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 18:14:35,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:35,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.65 MB 2025-02-14 18:14:35,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26928.04 MB 2025-02-14 18:14:35,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:14:35,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27076.33 MB 2025-02-14 18:14:35,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34626.08 MB 2025-02-14 18:14:35,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 18:14:35,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32472.33 MB 2025-02-14 18:14:35,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:14:35,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:14:35,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:14:35,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:35,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28461.59 MB 2025-02-14 18:14:35,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29228.59 MB 2025-02-14 18:14:35,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:14:35,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34626.08 MB 2025-02-14 18:14:35,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35041.31 MB 2025-02-14 18:14:35,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:14:35,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29936.38 MB 2025-02-14 18:14:35,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:14:35,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:14:35,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:14:35,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:35,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29641.48 MB 2025-02-14 18:14:35,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29870.19 MB 2025-02-14 18:14:35,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-14 18:14:35,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35041.31 MB 2025-02-14 18:14:35,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35041.31 MB 2025-02-14 18:14:35,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:14:35,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30107.30 MB 2025-02-14 18:14:35,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:14:35,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:14:35,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.66 seconds 2025-02-14 18:14:35,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:35,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17316.84 MB 2025-02-14 18:14:35,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30070.82 MB 2025-02-14 18:14:35,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12753.99 MB 2025-02-14 18:14:35,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54744.06 MB 2025-02-14 18:14:35,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35041.31 MB 2025-02-14 18:14:35,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19702.74 MB 2025-02-14 18:14:35,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30107.30 MB 2025-02-14 18:14:35,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:14:35,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:14:35,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:14:35,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:35,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19306.97 MB 2025-02-14 18:14:35,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22314.37 MB 2025-02-14 18:14:35,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3007.40 MB 2025-02-14 18:14:35,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35041.31 MB 2025-02-14 18:14:35,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35041.31 MB 2025-02-14 18:14:35,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:14:35,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22615.07 MB 2025-02-14 18:14:35,568 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 18:14:35,568 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:14:35,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:14:35,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:14:35,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:14:35,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:14:35,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22314.37 MB 2025-02-14 18:14:35,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30735.15 MB 2025-02-14 18:14:35,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 18:14:35,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35041.31 MB 2025-02-14 18:14:35,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43413.14 MB 2025-02-14 18:14:35,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 18:14:35,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30735.15 MB 2025-02-14 18:14:35,733 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 18:14:35,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:14:35,735 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:14:35,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:14:35,736 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:14:35,740 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:14:35,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:14:35,741 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:14:35,741 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:15:49,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:15:49,272 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:15:49,277 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:15:49,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:15:49,282 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1912, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:15:49,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:15:49,283 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1912, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:16:18,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:16:18,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:16:18,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.34 seconds 2025-02-14 18:16:18,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:18,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26291.83 MB 2025-02-14 18:16:18,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33059.34 MB 2025-02-14 18:16:18,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6767.51 MB 2025-02-14 18:16:18,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51784.97 MB 2025-02-14 18:16:18,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36607.89 MB 2025-02-14 18:16:18,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15177.09 MB 2025-02-14 18:16:18,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41879.30 MB 2025-02-14 18:16:18,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:16:18,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:16:18,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 18:16:18,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:18,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33059.34 MB 2025-02-14 18:16:18,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25718.79 MB 2025-02-14 18:16:18,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7340.55 MB 2025-02-14 18:16:18,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36607.89 MB 2025-02-14 18:16:18,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60723.04 MB 2025-02-14 18:16:18,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24115.15 MB 2025-02-14 18:16:18,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50910.12 MB 2025-02-14 18:16:20,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:16:20,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:16:20,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 18:16:20,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:20,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25718.79 MB 2025-02-14 18:16:20,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26249.63 MB 2025-02-14 18:16:20,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:16:20,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60723.04 MB 2025-02-14 18:16:20,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31964.79 MB 2025-02-14 18:16:20,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28758.25 MB 2025-02-14 18:16:20,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30228.18 MB 2025-02-14 18:16:20,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:16:20,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:16:20,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:16:20,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:20,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26249.63 MB 2025-02-14 18:16:20,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28139.17 MB 2025-02-14 18:16:20,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:16:20,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31964.79 MB 2025-02-14 18:16:20,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31964.79 MB 2025-02-14 18:16:20,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:16:20,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29556.60 MB 2025-02-14 18:16:20,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:16:20,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:16:20,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:16:20,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:20,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28139.17 MB 2025-02-14 18:16:20,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30381.02 MB 2025-02-14 18:16:20,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:16:20,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31964.79 MB 2025-02-14 18:16:20,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37627.10 MB 2025-02-14 18:16:20,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:16:20,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35925.30 MB 2025-02-14 18:16:20,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:16:20,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:16:20,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:16:20,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:20,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26249.63 MB 2025-02-14 18:16:20,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30381.02 MB 2025-02-14 18:16:20,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:16:20,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31964.79 MB 2025-02-14 18:16:20,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37627.10 MB 2025-02-14 18:16:20,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:16:20,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35925.30 MB 2025-02-14 18:16:21,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:16:21,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:16:21,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:16:21,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:21,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31914.57 MB 2025-02-14 18:16:21,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32681.57 MB 2025-02-14 18:16:21,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:16:21,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37627.10 MB 2025-02-14 18:16:21,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38044.43 MB 2025-02-14 18:16:21,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:16:21,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33389.36 MB 2025-02-14 18:16:21,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:16:21,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:16:21,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:16:21,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:21,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33094.46 MB 2025-02-14 18:16:21,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33323.00 MB 2025-02-14 18:16:21,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-14 18:16:21,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38044.43 MB 2025-02-14 18:16:21,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38044.43 MB 2025-02-14 18:16:21,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:16:21,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33538.68 MB 2025-02-14 18:16:21,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:16:21,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:16:21,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.82 seconds 2025-02-14 18:16:21,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:21,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19630.27 MB 2025-02-14 18:16:21,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33523.48 MB 2025-02-14 18:16:21,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13893.21 MB 2025-02-14 18:16:21,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51784.97 MB 2025-02-14 18:16:21,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38044.43 MB 2025-02-14 18:16:21,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13740.54 MB 2025-02-14 18:16:21,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33538.68 MB 2025-02-14 18:16:21,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:16:21,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:16:21,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:16:21,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:21,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33523.48 MB 2025-02-14 18:16:21,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24625.16 MB 2025-02-14 18:16:21,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8898.32 MB 2025-02-14 18:16:21,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38044.43 MB 2025-02-14 18:16:21,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38044.43 MB 2025-02-14 18:16:21,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:16:21,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36027.47 MB 2025-02-14 18:16:21,395 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 18:16:21,395 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:16:21,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:16:21,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:16:21,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:16:21,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:16:21,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24625.16 MB 2025-02-14 18:16:21,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33038.68 MB 2025-02-14 18:16:21,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 18:16:21,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38044.43 MB 2025-02-14 18:16:21,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46407.88 MB 2025-02-14 18:16:21,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 18:16:21,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33038.68 MB 2025-02-14 18:16:21,559 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 18:16:21,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:16:21,561 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:16:21,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:16:21,562 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:16:21,567 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:16:21,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:16:21,568 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:16:21,568 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:17:15,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:17:15,087 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:17:15,091 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:17:15,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:17:15,095 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1587, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:17:15,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:17:15,096 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1587, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:17:39,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:17:39,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:17:39,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.43 seconds 2025-02-14 18:17:39,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:39,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24027.18 MB 2025-02-14 18:17:39,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29643.48 MB 2025-02-14 18:17:39,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5616.30 MB 2025-02-14 18:17:39,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54771.32 MB 2025-02-14 18:17:39,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35450.26 MB 2025-02-14 18:17:39,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19321.06 MB 2025-02-14 18:17:39,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38482.19 MB 2025-02-14 18:17:39,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:17:39,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:17:39,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:17:39,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:39,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29643.48 MB 2025-02-14 18:17:39,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24029.22 MB 2025-02-14 18:17:39,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5614.26 MB 2025-02-14 18:17:39,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35450.26 MB 2025-02-14 18:17:39,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53016.00 MB 2025-02-14 18:17:39,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17565.75 MB 2025-02-14 18:17:39,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45395.20 MB 2025-02-14 18:17:41,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:17:41,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:17:41,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 18:17:41,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:41,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24029.22 MB 2025-02-14 18:17:41,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24560.06 MB 2025-02-14 18:17:41,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:17:41,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53016.00 MB 2025-02-14 18:17:41,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31956.40 MB 2025-02-14 18:17:41,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21059.60 MB 2025-02-14 18:17:41,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28539.43 MB 2025-02-14 18:17:41,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:17:41,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:17:41,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:17:41,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:41,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24560.06 MB 2025-02-14 18:17:41,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26449.38 MB 2025-02-14 18:17:41,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.32 MB 2025-02-14 18:17:41,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31956.40 MB 2025-02-14 18:17:41,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31956.40 MB 2025-02-14 18:17:41,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:17:41,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27866.81 MB 2025-02-14 18:17:41,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:17:41,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:17:41,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:17:41,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:41,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26449.38 MB 2025-02-14 18:17:41,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28691.23 MB 2025-02-14 18:17:41,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:17:41,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31956.40 MB 2025-02-14 18:17:41,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36203.13 MB 2025-02-14 18:17:41,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 18:17:41,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34235.52 MB 2025-02-14 18:17:41,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:17:41,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:17:41,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:17:41,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:41,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24560.06 MB 2025-02-14 18:17:41,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28691.23 MB 2025-02-14 18:17:41,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.17 MB 2025-02-14 18:17:41,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31956.40 MB 2025-02-14 18:17:41,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36203.13 MB 2025-02-14 18:17:41,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 18:17:41,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34235.52 MB 2025-02-14 18:17:41,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:17:41,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:17:41,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:17:41,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:41,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30224.78 MB 2025-02-14 18:17:41,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30991.78 MB 2025-02-14 18:17:41,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:17:41,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36203.13 MB 2025-02-14 18:17:41,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36620.47 MB 2025-02-14 18:17:41,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:17:41,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31699.57 MB 2025-02-14 18:17:41,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:17:41,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:17:41,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:17:41,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:41,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31404.67 MB 2025-02-14 18:17:41,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31633.95 MB 2025-02-14 18:17:41,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.28 MB 2025-02-14 18:17:41,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36620.47 MB 2025-02-14 18:17:41,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36620.47 MB 2025-02-14 18:17:41,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:17:41,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.99 MB 2025-02-14 18:17:41,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:17:41,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:17:41,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.86 seconds 2025-02-14 18:17:41,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:41,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18497.94 MB 2025-02-14 18:17:41,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31835.02 MB 2025-02-14 18:17:41,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13337.08 MB 2025-02-14 18:17:41,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54771.32 MB 2025-02-14 18:17:41,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36620.47 MB 2025-02-14 18:17:41,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18150.85 MB 2025-02-14 18:17:41,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.99 MB 2025-02-14 18:17:42,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:17:42,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:17:42,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:17:42,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:42,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31835.02 MB 2025-02-14 18:17:42,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23502.33 MB 2025-02-14 18:17:42,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8332.69 MB 2025-02-14 18:17:42,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36620.47 MB 2025-02-14 18:17:42,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36620.47 MB 2025-02-14 18:17:42,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:17:42,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34346.69 MB 2025-02-14 18:17:42,247 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:17:42,247 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:17:42,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:17:42,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:17:42,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:17:42,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:17:42,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23502.33 MB 2025-02-14 18:17:42,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31941.35 MB 2025-02-14 18:17:42,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:17:42,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36620.47 MB 2025-02-14 18:17:42,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45011.17 MB 2025-02-14 18:17:42,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:17:42,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31941.35 MB 2025-02-14 18:17:42,410 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:17:42,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:17:42,411 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:17:42,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:17:42,412 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:17:42,417 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:17:42,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:17:42,418 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:17:42,418 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:18:31,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:18:31,982 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:18:31,987 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:18:31,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:18:31,991 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:18:31,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:18:31,992 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:18:49,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:18:49,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:18:49,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.51 seconds 2025-02-14 18:18:49,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:49,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20884.54 MB 2025-02-14 18:18:49,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24904.78 MB 2025-02-14 18:18:49,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4020.24 MB 2025-02-14 18:18:49,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57596.18 MB 2025-02-14 18:18:49,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29672.60 MB 2025-02-14 18:18:49,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27923.58 MB 2025-02-14 18:18:49,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33753.29 MB 2025-02-14 18:18:49,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:18:49,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:18:49,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:18:49,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:49,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24904.78 MB 2025-02-14 18:18:49,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21683.56 MB 2025-02-14 18:18:49,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3221.22 MB 2025-02-14 18:18:49,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29672.60 MB 2025-02-14 18:18:49,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44086.33 MB 2025-02-14 18:18:49,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14413.73 MB 2025-02-14 18:18:49,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37165.85 MB 2025-02-14 18:18:51,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:18:51,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:18:51,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 18:18:51,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:51,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21683.56 MB 2025-02-14 18:18:51,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22214.40 MB 2025-02-14 18:18:51,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:18:51,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44086.33 MB 2025-02-14 18:18:51,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27067.94 MB 2025-02-14 18:18:51,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17018.39 MB 2025-02-14 18:18:51,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.95 MB 2025-02-14 18:18:51,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:18:51,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:18:51,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:18:51,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:51,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22214.40 MB 2025-02-14 18:18:51,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24103.94 MB 2025-02-14 18:18:51,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:18:51,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27067.94 MB 2025-02-14 18:18:51,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28011.66 MB 2025-02-14 18:18:51,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 18:18:51,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25521.36 MB 2025-02-14 18:18:51,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:18:51,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:18:51,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:18:51,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:51,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24103.94 MB 2025-02-14 18:18:51,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26345.79 MB 2025-02-14 18:18:51,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:18:51,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28011.66 MB 2025-02-14 18:18:51,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34145.83 MB 2025-02-14 18:18:51,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:18:51,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31890.07 MB 2025-02-14 18:18:51,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:18:51,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:18:51,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:18:51,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:51,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22214.40 MB 2025-02-14 18:18:51,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26345.79 MB 2025-02-14 18:18:51,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:18:51,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27067.94 MB 2025-02-14 18:18:51,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34145.83 MB 2025-02-14 18:18:51,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 18:18:51,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31890.07 MB 2025-02-14 18:18:51,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:18:51,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:18:51,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:18:51,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:51,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27879.33 MB 2025-02-14 18:18:51,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28646.34 MB 2025-02-14 18:18:51,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:18:51,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34145.83 MB 2025-02-14 18:18:51,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34561.06 MB 2025-02-14 18:18:51,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:18:51,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29354.12 MB 2025-02-14 18:18:51,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:18:51,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:18:51,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:18:51,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:51,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29059.23 MB 2025-02-14 18:18:51,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29287.68 MB 2025-02-14 18:18:51,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-14 18:18:51,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34561.06 MB 2025-02-14 18:18:51,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34561.06 MB 2025-02-14 18:18:51,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:18:51,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29520.83 MB 2025-02-14 18:18:51,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:18:51,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:18:51,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.92 seconds 2025-02-14 18:18:51,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:51,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16926.62 MB 2025-02-14 18:18:51,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29488.71 MB 2025-02-14 18:18:51,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12562.08 MB 2025-02-14 18:18:51,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57596.18 MB 2025-02-14 18:18:51,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34561.06 MB 2025-02-14 18:18:51,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23035.12 MB 2025-02-14 18:18:51,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29520.83 MB 2025-02-14 18:18:52,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:18:52,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:18:52,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:18:52,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:52,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29488.71 MB 2025-02-14 18:18:52,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21930.25 MB 2025-02-14 18:18:52,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7558.46 MB 2025-02-14 18:18:52,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34561.06 MB 2025-02-14 18:18:52,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34561.06 MB 2025-02-14 18:18:52,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:18:52,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31999.76 MB 2025-02-14 18:18:52,203 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 18:18:52,204 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:18:52,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:18:52,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:18:52,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:18:52,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:18:52,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21930.25 MB 2025-02-14 18:18:52,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30367.72 MB 2025-02-14 18:18:52,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 18:18:52,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34561.06 MB 2025-02-14 18:18:52,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42949.67 MB 2025-02-14 18:18:52,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 18:18:52,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30367.72 MB 2025-02-14 18:18:52,367 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 18:18:52,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:18:52,368 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:18:52,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:18:52,369 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:18:52,374 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:18:52,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:18:52,375 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:18:52,375 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:19:02,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:02,637 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:19:02,642 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:19:02,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:02,646 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1081, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:19:02,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:02,647 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1081, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:19:19,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:19:19,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:19:19,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.82 seconds 2025-02-14 18:19:19,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:19,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20501.29 MB 2025-02-14 18:19:19,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24326.89 MB 2025-02-14 18:19:19,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3825.60 MB 2025-02-14 18:19:19,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51338.28 MB 2025-02-14 18:19:19,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29479.67 MB 2025-02-14 18:19:19,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21858.62 MB 2025-02-14 18:19:19,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33143.55 MB 2025-02-14 18:19:19,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:19:19,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:19:19,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:19:19,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:19,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24326.89 MB 2025-02-14 18:19:19,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21397.63 MB 2025-02-14 18:19:19,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2929.25 MB 2025-02-14 18:19:19,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29479.67 MB 2025-02-14 18:19:19,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40198.21 MB 2025-02-14 18:19:19,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10718.54 MB 2025-02-14 18:19:19,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35698.44 MB 2025-02-14 18:19:21,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:19:21,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:19:21,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 18:19:21,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:21,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21397.63 MB 2025-02-14 18:19:21,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21928.47 MB 2025-02-14 18:19:21,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:19:21,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40198.21 MB 2025-02-14 18:19:21,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27067.94 MB 2025-02-14 18:19:21,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13130.27 MB 2025-02-14 18:19:21,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25907.02 MB 2025-02-14 18:19:21,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:19:21,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:19:21,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:19:21,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:21,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.47 MB 2025-02-14 18:19:21,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23818.01 MB 2025-02-14 18:19:21,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:19:21,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27067.94 MB 2025-02-14 18:19:21,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27067.94 MB 2025-02-14 18:19:21,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:21,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25235.44 MB 2025-02-14 18:19:21,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:19:21,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:19:21,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:19:21,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:21,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23818.01 MB 2025-02-14 18:19:21,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26059.86 MB 2025-02-14 18:19:21,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:19:21,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27067.94 MB 2025-02-14 18:19:21,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33202.11 MB 2025-02-14 18:19:21,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:19:21,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31604.15 MB 2025-02-14 18:19:21,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:19:21,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:19:21,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:19:21,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:21,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.47 MB 2025-02-14 18:19:21,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26059.86 MB 2025-02-14 18:19:21,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:19:21,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27067.94 MB 2025-02-14 18:19:21,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33202.11 MB 2025-02-14 18:19:21,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:19:21,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31604.15 MB 2025-02-14 18:19:21,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:19:21,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:19:21,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 18:19:21,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:21,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27593.41 MB 2025-02-14 18:19:21,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28360.41 MB 2025-02-14 18:19:21,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:19:21,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33202.11 MB 2025-02-14 18:19:21,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33617.35 MB 2025-02-14 18:19:21,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:19:21,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29068.20 MB 2025-02-14 18:19:21,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:19:21,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:19:21,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:19:21,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:21,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28773.30 MB 2025-02-14 18:19:21,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29002.33 MB 2025-02-14 18:19:21,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.04 MB 2025-02-14 18:19:21,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33617.35 MB 2025-02-14 18:19:21,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33617.35 MB 2025-02-14 18:19:21,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:21,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29203.49 MB 2025-02-14 18:19:21,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:19:21,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:19:21,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.26 seconds 2025-02-14 18:19:21,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:21,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16735.00 MB 2025-02-14 18:19:21,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29203.41 MB 2025-02-14 18:19:21,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12468.41 MB 2025-02-14 18:19:21,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51338.28 MB 2025-02-14 18:19:21,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33617.35 MB 2025-02-14 18:19:21,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17720.93 MB 2025-02-14 18:19:21,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29203.49 MB 2025-02-14 18:19:22,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:19:22,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:19:22,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:19:22,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:22,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29203.41 MB 2025-02-14 18:19:22,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21739.39 MB 2025-02-14 18:19:22,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7464.02 MB 2025-02-14 18:19:22,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33617.35 MB 2025-02-14 18:19:22,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33617.35 MB 2025-02-14 18:19:22,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:22,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31715.07 MB 2025-02-14 18:19:22,192 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:19:22,192 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:19:22,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:19:22,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:19:22,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:19:22,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:22,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21739.39 MB 2025-02-14 18:19:22,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30178.41 MB 2025-02-14 18:19:22,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:19:22,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33617.35 MB 2025-02-14 18:19:22,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42008.05 MB 2025-02-14 18:19:22,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:19:22,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30178.41 MB 2025-02-14 18:19:22,355 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:19:22,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:22,356 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:19:22,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:22,357 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:19:22,362 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:19:22,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:22,363 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:19:22,363 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:19:52,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:52,454 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:19:52,459 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:19:52,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:52,462 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 173, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:19:52,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:52,463 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 173, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:19:55,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:19:55,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:19:55,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.69 seconds 2025-02-14 18:19:55,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:55,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14174.20 MB 2025-02-14 18:19:55,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14786.44 MB 2025-02-14 18:19:55,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 612.24 MB 2025-02-14 18:19:55,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54593.06 MB 2025-02-14 18:19:55,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22768.78 MB 2025-02-14 18:19:55,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31824.28 MB 2025-02-14 18:19:55,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23645.57 MB 2025-02-14 18:19:55,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:19:55,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:19:55,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:19:55,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:55,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14786.44 MB 2025-02-14 18:19:55,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15083.06 MB 2025-02-14 18:19:55,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 296.63 MB 2025-02-14 18:19:55,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22768.78 MB 2025-02-14 18:19:55,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22768.78 MB 2025-02-14 18:19:55,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:55,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17262.47 MB 2025-02-14 18:19:56,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:19:56,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:19:56,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.83 seconds 2025-02-14 18:19:56,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15083.06 MB 2025-02-14 18:19:56,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15312.65 MB 2025-02-14 18:19:56,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.59 MB 2025-02-14 18:19:56,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22768.78 MB 2025-02-14 18:19:56,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22296.92 MB 2025-02-14 18:19:56,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 18:19:56,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19252.71 MB 2025-02-14 18:19:56,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:19:56,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:19:56,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:19:56,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.59 MB 2025-02-14 18:19:56,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16129.61 MB 2025-02-14 18:19:56,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.03 MB 2025-02-14 18:19:56,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22296.92 MB 2025-02-14 18:19:56,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22296.92 MB 2025-02-14 18:19:56,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:56,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16742.65 MB 2025-02-14 18:19:56,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:19:56,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:19:56,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:19:56,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16129.61 MB 2025-02-14 18:19:56,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17099.25 MB 2025-02-14 18:19:56,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 969.64 MB 2025-02-14 18:19:56,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22296.92 MB 2025-02-14 18:19:56,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22296.92 MB 2025-02-14 18:19:56,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:56,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19497.12 MB 2025-02-14 18:19:56,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:19:56,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:19:56,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:19:56,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.59 MB 2025-02-14 18:19:56,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17099.25 MB 2025-02-14 18:19:56,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1786.66 MB 2025-02-14 18:19:56,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22296.92 MB 2025-02-14 18:19:56,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22296.92 MB 2025-02-14 18:19:56,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:56,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19497.12 MB 2025-02-14 18:19:56,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:19:56,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:19:56,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:19:56,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17762.51 MB 2025-02-14 18:19:56,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18094.24 MB 2025-02-14 18:19:56,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 331.73 MB 2025-02-14 18:19:56,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22296.92 MB 2025-02-14 18:19:56,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22473.08 MB 2025-02-14 18:19:56,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 176.16 MB 2025-02-14 18:19:56,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18407.22 MB 2025-02-14 18:19:56,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:19:56,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:19:56,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:19:56,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18272.82 MB 2025-02-14 18:19:56,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18487.01 MB 2025-02-14 18:19:56,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.20 MB 2025-02-14 18:19:56,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22473.08 MB 2025-02-14 18:19:56,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22475.18 MB 2025-02-14 18:19:56,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 18:19:56,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18510.67 MB 2025-02-14 18:19:56,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:19:56,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:19:56,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-14 18:19:56,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13571.45 MB 2025-02-14 18:19:56,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18688.09 MB 2025-02-14 18:19:56,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5116.63 MB 2025-02-14 18:19:56,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54593.06 MB 2025-02-14 18:19:56,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22475.18 MB 2025-02-14 18:19:56,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32117.88 MB 2025-02-14 18:19:56,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18688.09 MB 2025-02-14 18:19:56,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:19:56,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:19:56,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:19:56,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18688.09 MB 2025-02-14 18:19:56,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17504.56 MB 2025-02-14 18:19:56,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1183.52 MB 2025-02-14 18:19:56,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22475.18 MB 2025-02-14 18:19:56,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22475.18 MB 2025-02-14 18:19:56,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:19:56,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18923.23 MB 2025-02-14 18:19:56,473 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:19:56,473 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:19:56,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:19:56,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:19:56,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:19:56,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:19:56,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17504.56 MB 2025-02-14 18:19:56,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25943.58 MB 2025-02-14 18:19:56,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:19:56,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22475.18 MB 2025-02-14 18:19:56,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30865.88 MB 2025-02-14 18:19:56,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:19:56,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25943.58 MB 2025-02-14 18:19:56,642 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:19:56,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:56,643 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:19:56,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:56,644 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:19:56,649 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:19:56,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:19:56,650 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:19:56,650 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:20:49,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:20:49,755 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:20:49,760 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:20:49,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:20:49,764 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 712, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:20:49,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:20:49,765 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 712, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:21:00,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:21:00,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:21:00,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.96 seconds 2025-02-14 18:21:00,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:00,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17930.04 MB 2025-02-14 18:21:00,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20450.81 MB 2025-02-14 18:21:00,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2520.78 MB 2025-02-14 18:21:00,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 18:21:00,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24815.60 MB 2025-02-14 18:21:00,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18635.29 MB 2025-02-14 18:21:00,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29439.84 MB 2025-02-14 18:21:00,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:21:00,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:21:00,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:21:00,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:00,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20450.81 MB 2025-02-14 18:21:00,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19479.32 MB 2025-02-14 18:21:00,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -971.50 MB 2025-02-14 18:21:00,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24815.60 MB 2025-02-14 18:21:00,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33332.13 MB 2025-02-14 18:21:00,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8516.53 MB 2025-02-14 18:21:00,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29579.86 MB 2025-02-14 18:21:02,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:21:02,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:21:02,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:21:02,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:02,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19479.32 MB 2025-02-14 18:21:02,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20010.16 MB 2025-02-14 18:21:02,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:21:02,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33332.13 MB 2025-02-14 18:21:02,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23710.40 MB 2025-02-14 18:21:02,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9621.73 MB 2025-02-14 18:21:02,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23988.71 MB 2025-02-14 18:21:02,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:21:02,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:21:02,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:21:02,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:02,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20010.16 MB 2025-02-14 18:21:02,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21899.69 MB 2025-02-14 18:21:02,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:21:02,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23710.40 MB 2025-02-14 18:21:02,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25597.84 MB 2025-02-14 18:21:02,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 18:21:02,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23317.12 MB 2025-02-14 18:21:02,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:21:02,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:21:02,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 18:21:02,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:02,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21899.69 MB 2025-02-14 18:21:02,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24141.55 MB 2025-02-14 18:21:02,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:21:02,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25597.84 MB 2025-02-14 18:21:02,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31732.01 MB 2025-02-14 18:21:02,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:21:02,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29685.83 MB 2025-02-14 18:21:02,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:21:02,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:21:02,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 18:21:02,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:02,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20010.16 MB 2025-02-14 18:21:02,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24141.55 MB 2025-02-14 18:21:02,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:21:02,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23710.40 MB 2025-02-14 18:21:02,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31732.01 MB 2025-02-14 18:21:02,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 18:21:02,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29685.83 MB 2025-02-14 18:21:03,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:21:03,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:21:03,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:21:03,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:03,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25675.09 MB 2025-02-14 18:21:03,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26442.09 MB 2025-02-14 18:21:03,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:21:03,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31732.01 MB 2025-02-14 18:21:03,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32149.34 MB 2025-02-14 18:21:03,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:21:03,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27149.88 MB 2025-02-14 18:21:03,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:21:03,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:21:03,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:21:03,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:03,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26854.98 MB 2025-02-14 18:21:03,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27084.68 MB 2025-02-14 18:21:03,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.70 MB 2025-02-14 18:21:03,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32149.34 MB 2025-02-14 18:21:03,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32149.34 MB 2025-02-14 18:21:03,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:21:03,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27320.57 MB 2025-02-14 18:21:03,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:21:03,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:21:03,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.37 seconds 2025-02-14 18:21:03,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:03,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15449.37 MB 2025-02-14 18:21:03,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27285.75 MB 2025-02-14 18:21:03,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11836.38 MB 2025-02-14 18:21:03,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 18:21:03,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32149.34 MB 2025-02-14 18:21:03,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11301.55 MB 2025-02-14 18:21:03,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27320.57 MB 2025-02-14 18:21:03,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:21:03,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:21:03,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:21:03,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:03,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27285.75 MB 2025-02-14 18:21:03,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20453.76 MB 2025-02-14 18:21:03,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6831.99 MB 2025-02-14 18:21:03,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32149.34 MB 2025-02-14 18:21:03,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32149.34 MB 2025-02-14 18:21:03,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:21:03,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29797.42 MB 2025-02-14 18:21:03,418 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:21:03,418 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:21:03,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:21:03,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:21:03,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:21:03,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:03,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20453.76 MB 2025-02-14 18:21:03,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.78 MB 2025-02-14 18:21:03,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:21:03,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32149.34 MB 2025-02-14 18:21:03,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42639.29 MB 2025-02-14 18:21:03,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 18:21:03,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.78 MB 2025-02-14 18:21:03,581 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:21:03,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:03,582 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:21:03,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:03,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:21:03,588 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:21:03,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:03,589 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:21:03,589 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:21:36,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:36,240 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:21:36,247 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:21:36,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:36,253 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:21:36,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:36,255 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:21:55,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:21:55,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:21:55,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.00 seconds 2025-02-14 18:21:55,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:55,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21546.51 MB 2025-02-14 18:21:55,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25902.95 MB 2025-02-14 18:21:55,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4356.44 MB 2025-02-14 18:21:55,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55224.30 MB 2025-02-14 18:21:55,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35041.31 MB 2025-02-14 18:21:55,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20182.99 MB 2025-02-14 18:21:55,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34868.25 MB 2025-02-14 18:21:55,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:21:55,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:21:55,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:21:55,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:55,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25902.95 MB 2025-02-14 18:21:55,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22177.44 MB 2025-02-14 18:21:55,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3725.52 MB 2025-02-14 18:21:55,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35041.31 MB 2025-02-14 18:21:55,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43570.43 MB 2025-02-14 18:21:55,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8529.12 MB 2025-02-14 18:21:55,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38780.03 MB 2025-02-14 18:21:57,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:21:57,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:21:57,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:21:57,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22177.44 MB 2025-02-14 18:21:57,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22708.28 MB 2025-02-14 18:21:57,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:21:57,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43570.43 MB 2025-02-14 18:21:57,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30683.43 MB 2025-02-14 18:21:57,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12887.00 MB 2025-02-14 18:21:57,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26686.82 MB 2025-02-14 18:21:57,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:21:57,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:21:57,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:21:57,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22708.28 MB 2025-02-14 18:21:57,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24597.81 MB 2025-02-14 18:21:57,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:21:57,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30683.43 MB 2025-02-14 18:21:57,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30683.43 MB 2025-02-14 18:21:57,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:21:57,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26015.24 MB 2025-02-14 18:21:57,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:21:57,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:21:57,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:21:57,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24597.81 MB 2025-02-14 18:21:57,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26839.67 MB 2025-02-14 18:21:57,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:21:57,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30683.43 MB 2025-02-14 18:21:57,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34458.30 MB 2025-02-14 18:21:57,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 18:21:57,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32383.95 MB 2025-02-14 18:21:57,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:21:57,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:21:57,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:21:57,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22708.28 MB 2025-02-14 18:21:57,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26839.67 MB 2025-02-14 18:21:57,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:21:57,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30683.43 MB 2025-02-14 18:21:57,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34458.30 MB 2025-02-14 18:21:57,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 18:21:57,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32383.95 MB 2025-02-14 18:21:57,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:21:57,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:21:57,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:21:57,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28373.21 MB 2025-02-14 18:21:57,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29140.21 MB 2025-02-14 18:21:57,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:21:57,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34458.30 MB 2025-02-14 18:21:57,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34875.64 MB 2025-02-14 18:21:57,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:21:57,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29848.00 MB 2025-02-14 18:21:57,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:21:57,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:21:57,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:21:57,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29553.10 MB 2025-02-14 18:21:57,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29781.85 MB 2025-02-14 18:21:57,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.75 MB 2025-02-14 18:21:57,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34875.64 MB 2025-02-14 18:21:57,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34875.64 MB 2025-02-14 18:21:57,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:21:57,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30023.34 MB 2025-02-14 18:21:57,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:21:57,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:21:57,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.40 seconds 2025-02-14 18:21:57,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17257.61 MB 2025-02-14 18:21:57,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29982.41 MB 2025-02-14 18:21:57,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12724.80 MB 2025-02-14 18:21:57,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55224.30 MB 2025-02-14 18:21:57,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34875.64 MB 2025-02-14 18:21:57,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20348.67 MB 2025-02-14 18:21:57,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30023.34 MB 2025-02-14 18:21:57,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:21:57,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:21:57,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:21:57,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29982.41 MB 2025-02-14 18:21:57,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22254.00 MB 2025-02-14 18:21:57,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7728.41 MB 2025-02-14 18:21:57,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34875.64 MB 2025-02-14 18:21:57,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34875.64 MB 2025-02-14 18:21:57,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:21:57,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32487.62 MB 2025-02-14 18:21:57,945 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 18:21:57,946 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:21:57,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:21:57,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:21:57,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:21:57,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:21:57,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22254.00 MB 2025-02-14 18:21:57,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30671.12 MB 2025-02-14 18:21:57,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.12 MB 2025-02-14 18:21:57,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34875.64 MB 2025-02-14 18:21:57,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39059.46 MB 2025-02-14 18:21:57,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 18:21:57,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30671.12 MB 2025-02-14 18:21:58,108 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 18:21:58,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:58,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:21:58,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:58,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:21:58,115 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:21:58,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:21:58,116 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:21:58,116 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:22:15,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:15,201 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:22:15,206 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:22:15,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:15,209 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 666, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:22:15,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:15,210 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 666, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:22:25,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:22:25,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:22:25,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.30 seconds 2025-02-14 18:22:25,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:25,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17609.50 MB 2025-02-14 18:22:25,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19966.44 MB 2025-02-14 18:22:25,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2356.94 MB 2025-02-14 18:22:25,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47427.09 MB 2025-02-14 18:22:25,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26489.13 MB 2025-02-14 18:22:25,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20937.97 MB 2025-02-14 18:22:25,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.81 MB 2025-02-14 18:22:25,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:22:25,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:22:25,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 18:22:25,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:25,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19966.44 MB 2025-02-14 18:22:25,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19241.23 MB 2025-02-14 18:22:25,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -725.21 MB 2025-02-14 18:22:25,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26489.13 MB 2025-02-14 18:22:25,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31998.35 MB 2025-02-14 18:22:25,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5509.22 MB 2025-02-14 18:22:25,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28752.00 MB 2025-02-14 18:22:27,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:22:27,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:22:27,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:22:27,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:27,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19241.23 MB 2025-02-14 18:22:27,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19772.07 MB 2025-02-14 18:22:27,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:22:27,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31998.35 MB 2025-02-14 18:22:27,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28613.54 MB 2025-02-14 18:22:27,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3384.80 MB 2025-02-14 18:22:27,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23750.61 MB 2025-02-14 18:22:27,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:22:27,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:22:27,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:22:27,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:27,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19772.07 MB 2025-02-14 18:22:27,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21661.60 MB 2025-02-14 18:22:27,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:22:27,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28613.54 MB 2025-02-14 18:22:27,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28613.54 MB 2025-02-14 18:22:27,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:27,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23079.03 MB 2025-02-14 18:22:27,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:22:27,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:22:27,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:22:27,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:27,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21661.60 MB 2025-02-14 18:22:27,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23903.46 MB 2025-02-14 18:22:27,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:22:27,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28613.54 MB 2025-02-14 18:22:27,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31916.56 MB 2025-02-14 18:22:27,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 18:22:27,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29447.74 MB 2025-02-14 18:22:27,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:22:27,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:22:27,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:22:27,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:27,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19772.07 MB 2025-02-14 18:22:27,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23903.46 MB 2025-02-14 18:22:27,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:22:27,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28613.54 MB 2025-02-14 18:22:27,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31916.56 MB 2025-02-14 18:22:27,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 18:22:27,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29447.74 MB 2025-02-14 18:22:27,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:22:27,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:22:27,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:22:27,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:27,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25437.00 MB 2025-02-14 18:22:27,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26204.00 MB 2025-02-14 18:22:27,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:22:27,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31916.56 MB 2025-02-14 18:22:27,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32333.89 MB 2025-02-14 18:22:27,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:22:27,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26911.79 MB 2025-02-14 18:22:27,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:22:27,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:22:27,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:22:27,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:27,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26616.89 MB 2025-02-14 18:22:27,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26844.94 MB 2025-02-14 18:22:27,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-14 18:22:27,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32333.89 MB 2025-02-14 18:22:27,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32333.89 MB 2025-02-14 18:22:27,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:27,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27076.25 MB 2025-02-14 18:22:27,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:22:27,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:22:27,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.67 seconds 2025-02-14 18:22:27,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:27,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15289.10 MB 2025-02-14 18:22:27,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27045.92 MB 2025-02-14 18:22:27,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11756.81 MB 2025-02-14 18:22:27,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47427.09 MB 2025-02-14 18:22:27,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32333.89 MB 2025-02-14 18:22:27,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15093.20 MB 2025-02-14 18:22:27,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27076.25 MB 2025-02-14 18:22:28,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:22:28,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:22:28,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:22:28,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:28,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27045.92 MB 2025-02-14 18:22:28,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20291.97 MB 2025-02-14 18:22:28,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6753.95 MB 2025-02-14 18:22:28,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32333.89 MB 2025-02-14 18:22:28,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32333.89 MB 2025-02-14 18:22:28,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:28,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29556.35 MB 2025-02-14 18:22:28,164 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 18:22:28,164 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:22:28,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:22:28,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:22:28,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:22:28,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:28,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20291.97 MB 2025-02-14 18:22:28,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28726.82 MB 2025-02-14 18:22:28,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 18:22:28,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32333.89 MB 2025-02-14 18:22:28,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40720.40 MB 2025-02-14 18:22:28,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 18:22:28,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28726.82 MB 2025-02-14 18:22:28,330 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 18:22:28,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:28,332 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:22:28,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:28,333 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:22:28,338 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:22:28,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:28,339 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:22:28,339 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:22:37,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:37,839 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:22:37,845 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:22:37,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:37,848 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 319, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:22:37,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:37,849 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 319, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:22:42,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:22:42,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:22:42,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.95 seconds 2025-02-14 18:22:42,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:42,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15191.55 MB 2025-02-14 18:22:42,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16320.47 MB 2025-02-14 18:22:42,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1128.92 MB 2025-02-14 18:22:42,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53299.12 MB 2025-02-14 18:22:42,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 18:22:42,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25128.08 MB 2025-02-14 18:22:42,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25197.95 MB 2025-02-14 18:22:42,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:22:42,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:22:42,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:22:42,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:42,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16320.47 MB 2025-02-14 18:22:42,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16059.72 MB 2025-02-14 18:22:42,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -260.75 MB 2025-02-14 18:22:42,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 18:22:42,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 18:22:42,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:42,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19238.95 MB 2025-02-14 18:22:43,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:22:43,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:22:43,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.98 seconds 2025-02-14 18:22:43,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:43,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16059.72 MB 2025-02-14 18:22:43,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16330.45 MB 2025-02-14 18:22:43,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 270.73 MB 2025-02-14 18:22:43,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 18:22:43,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 18:22:43,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:43,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20314.31 MB 2025-02-14 18:22:43,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:22:43,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:22:43,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:22:43,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:43,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16330.45 MB 2025-02-14 18:22:43,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17293.88 MB 2025-02-14 18:22:43,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 963.43 MB 2025-02-14 18:22:43,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 18:22:43,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 18:22:43,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:43,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18016.77 MB 2025-02-14 18:22:43,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:22:43,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:22:43,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:22:43,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:43,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17293.88 MB 2025-02-14 18:22:43,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18437.26 MB 2025-02-14 18:22:43,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1143.38 MB 2025-02-14 18:22:43,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 18:22:43,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 18:22:43,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:43,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21264.81 MB 2025-02-14 18:22:43,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:22:43,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:22:43,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 18:22:43,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:43,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16330.45 MB 2025-02-14 18:22:43,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18437.26 MB 2025-02-14 18:22:43,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2106.81 MB 2025-02-14 18:22:43,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 18:22:43,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 18:22:43,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:43,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21264.81 MB 2025-02-14 18:22:44,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:22:44,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:22:44,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:22:44,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:44,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19219.36 MB 2025-02-14 18:22:44,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19610.53 MB 2025-02-14 18:22:44,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 391.17 MB 2025-02-14 18:22:44,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 18:22:44,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28378.66 MB 2025-02-14 18:22:44,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-14 18:22:44,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19975.65 MB 2025-02-14 18:22:44,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:22:44,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:22:44,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:22:44,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:44,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19821.11 MB 2025-02-14 18:22:44,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20034.41 MB 2025-02-14 18:22:44,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.30 MB 2025-02-14 18:22:44,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28378.66 MB 2025-02-14 18:22:44,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28378.66 MB 2025-02-14 18:22:44,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:44,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20080.09 MB 2025-02-14 18:22:44,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:22:44,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:22:44,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.16 seconds 2025-02-14 18:22:44,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:44,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14080.13 MB 2025-02-14 18:22:44,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20235.48 MB 2025-02-14 18:22:44,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6155.36 MB 2025-02-14 18:22:44,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53299.12 MB 2025-02-14 18:22:44,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28378.66 MB 2025-02-14 18:22:44,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24920.46 MB 2025-02-14 18:22:44,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20235.48 MB 2025-02-14 18:22:44,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:22:44,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:22:44,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:22:44,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:44,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15145.50 MB 2025-02-14 18:22:44,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18159.93 MB 2025-02-14 18:22:44,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.43 MB 2025-02-14 18:22:44,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28378.66 MB 2025-02-14 18:22:44,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28378.66 MB 2025-02-14 18:22:44,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:22:44,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.30 MB 2025-02-14 18:22:44,298 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:22:44,298 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:22:44,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:22:44,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:22:44,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:22:44,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:22:44,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18159.93 MB 2025-02-14 18:22:44,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26598.95 MB 2025-02-14 18:22:44,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:22:44,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28378.66 MB 2025-02-14 18:22:44,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36769.37 MB 2025-02-14 18:22:44,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:22:44,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26598.95 MB 2025-02-14 18:22:44,460 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:22:44,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:44,461 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:22:44,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:44,462 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:22:44,467 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:22:44,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:22:44,468 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:22:44,468 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:24:06,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:06,739 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:24:06,744 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:24:06,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:06,748 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:24:06,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:06,750 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:24:09,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:24:09,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:24:09,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.75 seconds 2025-02-14 18:24:09,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:09,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-14 18:24:09,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-14 18:24:09,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-14 18:24:09,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49354.38 MB 2025-02-14 18:24:09,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 18:24:09,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27686.60 MB 2025-02-14 18:24:09,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23687.38 MB 2025-02-14 18:24:09,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:24:09,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:24:09,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:24:09,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:09,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-14 18:24:09,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15037.00 MB 2025-02-14 18:24:09,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 187.52 MB 2025-02-14 18:24:09,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21667.77 MB 2025-02-14 18:24:09,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 18:24:09,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:24:09,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17135.62 MB 2025-02-14 18:24:10,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:24:10,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:24:10,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 18:24:10,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15037.00 MB 2025-02-14 18:24:10,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15251.99 MB 2025-02-14 18:24:10,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-14 18:24:10,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21667.77 MB 2025-02-14 18:24:10,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 18:24:10,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:24:10,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19206.65 MB 2025-02-14 18:24:10,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:24:10,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:24:10,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:24:10,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15251.93 MB 2025-02-14 18:24:10,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16017.00 MB 2025-02-14 18:24:10,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-14 18:24:10,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21667.77 MB 2025-02-14 18:24:10,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 18:24:10,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:24:10,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16591.07 MB 2025-02-14 18:24:10,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:24:10,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:24:10,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:24:10,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16017.00 MB 2025-02-14 18:24:10,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16924.99 MB 2025-02-14 18:24:10,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-14 18:24:10,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21667.77 MB 2025-02-14 18:24:10,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 18:24:10,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:24:10,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19170.39 MB 2025-02-14 18:24:10,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:24:10,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:24:10,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:24:10,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15251.93 MB 2025-02-14 18:24:10,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16924.99 MB 2025-02-14 18:24:10,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-14 18:24:10,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21667.77 MB 2025-02-14 18:24:10,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 18:24:10,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:24:10,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19170.39 MB 2025-02-14 18:24:10,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:24:10,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:24:10,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:24:10,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17546.08 MB 2025-02-14 18:24:10,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17856.71 MB 2025-02-14 18:24:10,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.64 MB 2025-02-14 18:24:10,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21667.77 MB 2025-02-14 18:24:10,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21835.55 MB 2025-02-14 18:24:10,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 18:24:10,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18152.39 MB 2025-02-14 18:24:10,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:24:10,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:24:10,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:24:10,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18023.94 MB 2025-02-14 18:24:10,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18227.67 MB 2025-02-14 18:24:10,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.73 MB 2025-02-14 18:24:10,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21835.55 MB 2025-02-14 18:24:10,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21839.74 MB 2025-02-14 18:24:10,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 18:24:10,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18248.66 MB 2025-02-14 18:24:10,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:24:10,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:24:10,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-14 18:24:10,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-14 18:24:10,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18428.35 MB 2025-02-14 18:24:10,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4835.99 MB 2025-02-14 18:24:10,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49354.38 MB 2025-02-14 18:24:10,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21839.74 MB 2025-02-14 18:24:10,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27514.63 MB 2025-02-14 18:24:10,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18428.35 MB 2025-02-14 18:24:10,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:24:10,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:24:10,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:24:10,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18428.35 MB 2025-02-14 18:24:10,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17467.13 MB 2025-02-14 18:24:10,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -961.22 MB 2025-02-14 18:24:10,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21839.74 MB 2025-02-14 18:24:10,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21839.74 MB 2025-02-14 18:24:10,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:24:10,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19130.24 MB 2025-02-14 18:24:10,755 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 18:24:10,755 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:24:10,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:24:10,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:24:10,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:24:10,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:24:10,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17467.13 MB 2025-02-14 18:24:10,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25889.46 MB 2025-02-14 18:24:10,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 18:24:10,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21839.74 MB 2025-02-14 18:24:10,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30213.67 MB 2025-02-14 18:24:10,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 18:24:10,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25889.46 MB 2025-02-14 18:24:10,922 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 18:24:10,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:10,924 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:24:10,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:10,925 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:24:10,930 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:24:10,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:10,931 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:24:10,931 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:24:55,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:55,714 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:24:55,719 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:24:55,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:55,724 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1762, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:24:55,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:24:55,725 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1762, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:25:22,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:25:22,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:25:22,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.12 seconds 2025-02-14 18:25:22,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:22,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25246.60 MB 2025-02-14 18:25:22,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31482.22 MB 2025-02-14 18:25:22,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6235.62 MB 2025-02-14 18:25:22,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42773.51 MB 2025-02-14 18:25:22,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36909.88 MB 2025-02-14 18:25:22,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5863.64 MB 2025-02-14 18:25:22,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40381.09 MB 2025-02-14 18:25:22,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:25:22,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:25:22,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:25:22,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:22,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31482.22 MB 2025-02-14 18:25:22,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24937.94 MB 2025-02-14 18:25:22,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6544.28 MB 2025-02-14 18:25:22,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36909.88 MB 2025-02-14 18:25:22,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57732.50 MB 2025-02-14 18:25:22,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20822.62 MB 2025-02-14 18:25:22,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48846.10 MB 2025-02-14 18:25:24,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:25:24,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:25:24,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 18:25:24,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:24,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24937.94 MB 2025-02-14 18:25:24,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25468.78 MB 2025-02-14 18:25:24,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:25:24,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57732.50 MB 2025-02-14 18:25:24,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-14 18:25:24,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25643.97 MB 2025-02-14 18:25:24,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29447.33 MB 2025-02-14 18:25:24,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:25:24,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:25:24,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:25:24,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:24,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25468.78 MB 2025-02-14 18:25:24,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27358.31 MB 2025-02-14 18:25:24,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:25:24,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 18:25:24,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-14 18:25:24,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:25:24,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28775.74 MB 2025-02-14 18:25:25,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:25:25,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:25:25,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:25:25,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:25,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27358.31 MB 2025-02-14 18:25:25,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.17 MB 2025-02-14 18:25:25,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:25:25,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 18:25:25,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37278.97 MB 2025-02-14 18:25:25,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:25:25,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35144.45 MB 2025-02-14 18:25:25,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:25:25,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:25:25,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:25:25,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:25,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25468.78 MB 2025-02-14 18:25:25,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.17 MB 2025-02-14 18:25:25,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:25:25,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 18:25:25,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37278.97 MB 2025-02-14 18:25:25,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:25:25,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35144.45 MB 2025-02-14 18:25:25,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:25:25,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:25:25,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:25:25,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:25,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31133.71 MB 2025-02-14 18:25:25,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31900.71 MB 2025-02-14 18:25:25,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:25:25,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37278.97 MB 2025-02-14 18:25:25,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37694.21 MB 2025-02-14 18:25:25,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:25:25,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32608.50 MB 2025-02-14 18:25:25,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:25:25,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:25:25,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:25:25,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:25,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32313.60 MB 2025-02-14 18:25:25,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32542.10 MB 2025-02-14 18:25:25,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.49 MB 2025-02-14 18:25:25,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37694.21 MB 2025-02-14 18:25:25,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37694.21 MB 2025-02-14 18:25:25,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:25:25,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32768.69 MB 2025-02-14 18:25:25,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:25:25,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:25:25,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.57 seconds 2025-02-14 18:25:25,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:25,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19107.66 MB 2025-02-14 18:25:25,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32742.51 MB 2025-02-14 18:25:25,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13634.85 MB 2025-02-14 18:25:25,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42773.51 MB 2025-02-14 18:25:25,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37694.21 MB 2025-02-14 18:25:25,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5079.30 MB 2025-02-14 18:25:25,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32768.69 MB 2025-02-14 18:25:25,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:25:25,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:25:25,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:25:25,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:25,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32742.51 MB 2025-02-14 18:25:25,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24101.83 MB 2025-02-14 18:25:25,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8640.67 MB 2025-02-14 18:25:25,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37694.21 MB 2025-02-14 18:25:25,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37694.21 MB 2025-02-14 18:25:25,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:25:25,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35245.88 MB 2025-02-14 18:25:25,580 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 18:25:25,580 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:25:25,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:25:25,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:25:25,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:25:25,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:25,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24101.83 MB 2025-02-14 18:25:25,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32512.65 MB 2025-02-14 18:25:25,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 18:25:25,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37694.21 MB 2025-02-14 18:25:25,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41875.93 MB 2025-02-14 18:25:25,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 18:25:25,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32512.65 MB 2025-02-14 18:25:25,747 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 18:25:25,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:25,748 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:25:25,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:25,749 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:25:25,754 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:25:25,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:25,755 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:25:25,755 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:25:36,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:36,873 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:25:36,881 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:25:36,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:36,887 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:25:36,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:36,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:25:55,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:25:55,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:25:55,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.03 seconds 2025-02-14 18:25:55,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:55,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-14 18:25:55,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-14 18:25:55,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 18:25:55,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50239.37 MB 2025-02-14 18:25:55,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30790.39 MB 2025-02-14 18:25:55,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19448.99 MB 2025-02-14 18:25:55,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-14 18:25:55,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:25:55,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:25:55,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 18:25:55,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:55,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-14 18:25:55,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20967.25 MB 2025-02-14 18:25:55,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4789.26 MB 2025-02-14 18:25:55,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30790.39 MB 2025-02-14 18:25:55,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30790.39 MB 2025-02-14 18:25:55,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:25:55,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29103.33 MB 2025-02-14 18:25:57,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:25:57,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:25:57,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.15 seconds 2025-02-14 18:25:57,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20967.25 MB 2025-02-14 18:25:57,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21283.10 MB 2025-02-14 18:25:57,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 315.85 MB 2025-02-14 18:25:57,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30790.39 MB 2025-02-14 18:25:57,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 18:25:57,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4307.55 MB 2025-02-14 18:25:57,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25221.83 MB 2025-02-14 18:25:57,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:25:57,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:25:57,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:25:57,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21283.10 MB 2025-02-14 18:25:57,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22407.10 MB 2025-02-14 18:25:57,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1124.00 MB 2025-02-14 18:25:57,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 18:25:57,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 18:25:57,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:25:57,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23250.47 MB 2025-02-14 18:25:57,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:25:57,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:25:57,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 18:25:57,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22407.10 MB 2025-02-14 18:25:57,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23741.03 MB 2025-02-14 18:25:57,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1333.93 MB 2025-02-14 18:25:57,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 18:25:57,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28449.96 MB 2025-02-14 18:25:57,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1967.13 MB 2025-02-14 18:25:57,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27041.68 MB 2025-02-14 18:25:57,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:25:57,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:25:57,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 18:25:57,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21283.10 MB 2025-02-14 18:25:57,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23741.03 MB 2025-02-14 18:25:57,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2457.93 MB 2025-02-14 18:25:57,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 18:25:57,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28449.96 MB 2025-02-14 18:25:57,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1967.13 MB 2025-02-14 18:25:57,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27041.68 MB 2025-02-14 18:25:57,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:25:57,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:25:57,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:25:57,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24653.49 MB 2025-02-14 18:25:57,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25110.38 MB 2025-02-14 18:25:57,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 456.89 MB 2025-02-14 18:25:57,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28449.96 MB 2025-02-14 18:25:57,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28695.33 MB 2025-02-14 18:25:57,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 245.37 MB 2025-02-14 18:25:57,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25531.51 MB 2025-02-14 18:25:57,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:25:57,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:25:57,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:25:57,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25356.05 MB 2025-02-14 18:25:57,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25577.18 MB 2025-02-14 18:25:57,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.13 MB 2025-02-14 18:25:57,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28695.33 MB 2025-02-14 18:25:57,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28695.33 MB 2025-02-14 18:25:57,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:25:57,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25636.79 MB 2025-02-14 18:25:57,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:25:57,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:25:57,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.49 seconds 2025-02-14 18:25:57,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-14 18:25:57,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25778.25 MB 2025-02-14 18:25:57,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8569.42 MB 2025-02-14 18:25:57,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50239.37 MB 2025-02-14 18:25:57,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28695.33 MB 2025-02-14 18:25:57,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21544.04 MB 2025-02-14 18:25:57,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25778.25 MB 2025-02-14 18:25:57,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:25:57,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:25:57,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:25:57,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25778.25 MB 2025-02-14 18:25:57,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28792.29 MB 2025-02-14 18:25:57,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 18:25:57,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28695.33 MB 2025-02-14 18:25:57,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29903.29 MB 2025-02-14 18:25:57,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1207.96 MB 2025-02-14 18:25:57,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29093.92 MB 2025-02-14 18:25:57,675 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:25:57,676 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:25:57,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:25:57,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:25:57,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:25:57,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:25:57,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21449.22 MB 2025-02-14 18:25:57,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29888.24 MB 2025-02-14 18:25:57,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:25:57,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29903.29 MB 2025-02-14 18:25:57,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40393.24 MB 2025-02-14 18:25:57,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 18:25:57,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29888.24 MB 2025-02-14 18:25:57,837 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:25:57,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:57,838 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:25:57,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:57,839 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:25:57,844 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:25:57,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:25:57,845 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:25:57,845 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:26:22,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:26:22,577 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:26:22,585 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:26:22,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:26:22,591 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 134, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:26:22,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:26:22,593 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 134, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:26:24,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:26:24,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:26:24,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.19 seconds 2025-02-14 18:26:24,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:24,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13902.44 MB 2025-02-14 18:26:24,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14376.66 MB 2025-02-14 18:26:24,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 474.22 MB 2025-02-14 18:26:24,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52978.25 MB 2025-02-14 18:26:24,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19518.19 MB 2025-02-14 18:26:24,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33460.06 MB 2025-02-14 18:26:24,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23373.81 MB 2025-02-14 18:26:24,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:26:24,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:26:24,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:26:24,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:24,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14376.66 MB 2025-02-14 18:26:24,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14606.42 MB 2025-02-14 18:26:24,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.76 MB 2025-02-14 18:26:24,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19518.19 MB 2025-02-14 18:26:24,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19518.19 MB 2025-02-14 18:26:24,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:26:24,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16280.13 MB 2025-02-14 18:26:25,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:26:25,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:26:25,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.69 seconds 2025-02-14 18:26:25,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:25,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.42 MB 2025-02-14 18:26:25,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14784.25 MB 2025-02-14 18:26:25,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-14 18:26:25,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19518.19 MB 2025-02-14 18:26:25,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19237.18 MB 2025-02-14 18:26:25,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -281.02 MB 2025-02-14 18:26:25,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18777.11 MB 2025-02-14 18:26:25,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:26:25,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:26:25,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:26:25,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:25,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.18 MB 2025-02-14 18:26:25,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15417.02 MB 2025-02-14 18:26:25,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-14 18:26:25,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19237.18 MB 2025-02-14 18:26:25,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19237.18 MB 2025-02-14 18:26:25,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:26:25,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15891.87 MB 2025-02-14 18:26:25,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:26:25,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:26:25,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:26:25,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:25,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15417.02 MB 2025-02-14 18:26:25,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16168.09 MB 2025-02-14 18:26:25,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-14 18:26:25,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19237.18 MB 2025-02-14 18:26:25,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19237.18 MB 2025-02-14 18:26:25,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:26:25,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18025.38 MB 2025-02-14 18:26:25,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:26:25,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:26:25,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:26:25,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:25,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.18 MB 2025-02-14 18:26:25,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16168.09 MB 2025-02-14 18:26:25,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-14 18:26:25,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19237.18 MB 2025-02-14 18:26:25,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19237.18 MB 2025-02-14 18:26:25,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:26:25,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18025.38 MB 2025-02-14 18:26:25,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:26:25,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:26:25,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:26:25,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:25,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16681.82 MB 2025-02-14 18:26:25,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16938.77 MB 2025-02-14 18:26:25,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-14 18:26:25,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19237.18 MB 2025-02-14 18:26:25,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19375.59 MB 2025-02-14 18:26:25,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 138.41 MB 2025-02-14 18:26:25,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17188.04 MB 2025-02-14 18:26:25,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:26:25,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:26:25,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:26:25,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:25,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17077.10 MB 2025-02-14 18:26:25,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17297.71 MB 2025-02-14 18:26:25,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.62 MB 2025-02-14 18:26:25,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19375.59 MB 2025-02-14 18:26:25,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19375.59 MB 2025-02-14 18:26:25,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:26:25,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17299.90 MB 2025-02-14 18:26:25,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:26:25,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:26:25,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.12 seconds 2025-02-14 18:26:25,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:25,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13435.57 MB 2025-02-14 18:26:25,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14170.41 MB 2025-02-14 18:26:25,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.84 MB 2025-02-14 18:26:25,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52978.25 MB 2025-02-14 18:26:25,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19375.59 MB 2025-02-14 18:26:25,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33602.67 MB 2025-02-14 18:26:25,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17498.42 MB 2025-02-14 18:26:26,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:26:26,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:26:26,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 18:26:26,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:26,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14170.41 MB 2025-02-14 18:26:26,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17178.92 MB 2025-02-14 18:26:26,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-14 18:26:26,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19375.59 MB 2025-02-14 18:26:26,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19375.59 MB 2025-02-14 18:26:26,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:26:26,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17479.73 MB 2025-02-14 18:26:26,030 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 18:26:26,031 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:26:26,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:26:26,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:26:26,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:26:26,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:26:26,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17178.92 MB 2025-02-14 18:26:26,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25602.12 MB 2025-02-14 18:26:26,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 18:26:26,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19375.59 MB 2025-02-14 18:26:26,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29846.67 MB 2025-02-14 18:26:26,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 18:26:26,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25602.12 MB 2025-02-14 18:26:26,291 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 18:26:26,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:26:26,293 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:26:26,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:26:26,295 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:26:26,303 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:26:26,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:26:26,306 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:26:26,306 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:27:16,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:27:16,781 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:27:16,787 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:27:16,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:27:16,791 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 657, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:27:16,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:27:16,792 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 657, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:27:26,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:27:26,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:27:26,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.11 seconds 2025-02-14 18:27:26,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:26,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22882.90 MB 2025-02-14 18:27:26,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25207.99 MB 2025-02-14 18:27:26,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2325.09 MB 2025-02-14 18:27:26,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38222.69 MB 2025-02-14 18:27:26,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29429.33 MB 2025-02-14 18:27:26,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8793.36 MB 2025-02-14 18:27:26,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34166.21 MB 2025-02-14 18:27:26,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:27:26,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:27:26,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 18:27:26,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:26,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25207.99 MB 2025-02-14 18:27:26,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19193.39 MB 2025-02-14 18:27:26,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6014.60 MB 2025-02-14 18:27:26,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29429.33 MB 2025-02-14 18:27:26,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32260.49 MB 2025-02-14 18:27:26,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 18:27:26,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27864.54 MB 2025-02-14 18:27:28,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:27:28,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:27:28,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:27:28,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:28,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19193.39 MB 2025-02-14 18:27:28,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19724.23 MB 2025-02-14 18:27:28,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:27:28,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32260.49 MB 2025-02-14 18:27:28,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27906.80 MB 2025-02-14 18:27:28,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4353.69 MB 2025-02-14 18:27:28,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23702.78 MB 2025-02-14 18:27:28,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:27:28,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:27:28,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:27:28,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:28,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19724.23 MB 2025-02-14 18:27:28,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21613.76 MB 2025-02-14 18:27:28,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:27:28,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 18:27:28,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27906.80 MB 2025-02-14 18:27:28,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:27:28,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23031.19 MB 2025-02-14 18:27:29,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:27:29,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:27:29,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:27:29,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:29,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21613.76 MB 2025-02-14 18:27:29,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23855.62 MB 2025-02-14 18:27:29,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:27:29,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 18:27:29,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32153.53 MB 2025-02-14 18:27:29,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 18:27:29,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29399.90 MB 2025-02-14 18:27:29,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:27:29,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:27:29,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 18:27:29,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:29,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19724.23 MB 2025-02-14 18:27:29,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23855.62 MB 2025-02-14 18:27:29,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:27:29,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 18:27:29,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32153.53 MB 2025-02-14 18:27:29,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 18:27:29,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29399.90 MB 2025-02-14 18:27:29,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:27:29,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:27:29,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 18:27:29,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:29,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25389.16 MB 2025-02-14 18:27:29,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26156.16 MB 2025-02-14 18:27:29,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:27:29,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32153.53 MB 2025-02-14 18:27:29,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32568.77 MB 2025-02-14 18:27:29,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:27:29,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26863.95 MB 2025-02-14 18:27:29,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:27:29,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:27:29,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:27:29,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:29,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26569.05 MB 2025-02-14 18:27:29,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26799.22 MB 2025-02-14 18:27:29,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.17 MB 2025-02-14 18:27:29,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32568.77 MB 2025-02-14 18:27:29,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32568.77 MB 2025-02-14 18:27:29,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:27:29,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26995.76 MB 2025-02-14 18:27:29,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:27:29,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:27:29,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.50 seconds 2025-02-14 18:27:29,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:29,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20593.86 MB 2025-02-14 18:27:29,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27000.29 MB 2025-02-14 18:27:29,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6406.43 MB 2025-02-14 18:27:29,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38222.69 MB 2025-02-14 18:27:29,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32568.77 MB 2025-02-14 18:27:29,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5653.92 MB 2025-02-14 18:27:29,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27000.29 MB 2025-02-14 18:27:29,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:27:29,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:27:29,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:27:29,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:29,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27000.29 MB 2025-02-14 18:27:29,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20262.14 MB 2025-02-14 18:27:29,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6738.16 MB 2025-02-14 18:27:29,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32568.77 MB 2025-02-14 18:27:29,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32568.77 MB 2025-02-14 18:27:29,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:27:29,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29511.96 MB 2025-02-14 18:27:29,585 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:27:29,586 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 18:27:29,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:27:29,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:27:29,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:27:29,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:27:29,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20262.14 MB 2025-02-14 18:27:29,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28701.16 MB 2025-02-14 18:27:29,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:27:29,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32568.77 MB 2025-02-14 18:27:29,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40959.48 MB 2025-02-14 18:27:29,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:27:29,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28701.16 MB 2025-02-14 18:27:29,758 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:27:29,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:27:29,759 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:27:29,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:27:29,760 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:27:29,765 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:27:29,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:27:29,766 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:27:29,766 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 18:28:17,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:28:17,914 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:28:17,919 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:28:17,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:28:17,923 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:28:17,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:28:17,924 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:28:36,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:28:36,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:28:36,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.64 seconds 2025-02-14 18:28:36,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:36,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21407.15 MB 2025-02-14 18:28:36,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25693.73 MB 2025-02-14 18:28:36,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4286.58 MB 2025-02-14 18:28:36,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53544.48 MB 2025-02-14 18:28:36,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30775.71 MB 2025-02-14 18:28:36,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22768.78 MB 2025-02-14 18:28:36,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34502.40 MB 2025-02-14 18:28:36,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:28:36,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:28:36,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:28:36,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:36,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25693.73 MB 2025-02-14 18:28:36,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22074.51 MB 2025-02-14 18:28:36,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3619.22 MB 2025-02-14 18:28:36,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30775.71 MB 2025-02-14 18:28:36,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44751.13 MB 2025-02-14 18:28:36,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13975.42 MB 2025-02-14 18:28:36,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38266.73 MB 2025-02-14 18:28:38,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:28:38,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:28:38,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 18:28:38,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:38,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22074.51 MB 2025-02-14 18:28:38,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22605.35 MB 2025-02-14 18:28:38,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:28:38,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44751.13 MB 2025-02-14 18:28:38,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30643.59 MB 2025-02-14 18:28:38,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14107.54 MB 2025-02-14 18:28:38,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26583.90 MB 2025-02-14 18:28:38,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:28:38,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:28:38,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:28:38,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:38,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22605.35 MB 2025-02-14 18:28:38,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24494.89 MB 2025-02-14 18:28:38,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:28:38,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30643.59 MB 2025-02-14 18:28:38,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30643.59 MB 2025-02-14 18:28:38,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:28:38,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25912.31 MB 2025-02-14 18:28:38,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:28:38,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:28:38,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:28:38,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:38,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24494.89 MB 2025-02-14 18:28:38,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26736.74 MB 2025-02-14 18:28:38,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:28:38,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30643.59 MB 2025-02-14 18:28:38,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 18:28:38,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 18:28:38,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32281.02 MB 2025-02-14 18:28:38,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:28:38,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:28:38,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:28:38,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:38,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22605.35 MB 2025-02-14 18:28:38,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26736.74 MB 2025-02-14 18:28:38,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:28:38,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30643.59 MB 2025-02-14 18:28:38,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 18:28:38,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 18:28:38,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32281.02 MB 2025-02-14 18:28:38,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:28:38,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:28:38,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 18:28:38,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:38,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28270.28 MB 2025-02-14 18:28:38,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29037.29 MB 2025-02-14 18:28:38,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:28:38,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34418.46 MB 2025-02-14 18:28:38,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34835.79 MB 2025-02-14 18:28:38,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:28:38,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29745.07 MB 2025-02-14 18:28:39,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:28:39,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:28:39,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:28:39,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:39,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29450.17 MB 2025-02-14 18:28:39,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29678.05 MB 2025-02-14 18:28:39,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.87 MB 2025-02-14 18:28:39,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34835.79 MB 2025-02-14 18:28:39,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34835.79 MB 2025-02-14 18:28:39,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:28:39,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29903.32 MB 2025-02-14 18:28:39,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:28:39,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:28:39,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.09 seconds 2025-02-14 18:28:39,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:39,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17187.93 MB 2025-02-14 18:28:39,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29878.63 MB 2025-02-14 18:28:39,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12690.70 MB 2025-02-14 18:28:39,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53544.48 MB 2025-02-14 18:28:39,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34835.79 MB 2025-02-14 18:28:39,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18708.69 MB 2025-02-14 18:28:39,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29903.32 MB 2025-02-14 18:28:39,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:28:39,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:28:39,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:28:39,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:39,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29878.63 MB 2025-02-14 18:28:39,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22184.70 MB 2025-02-14 18:28:39,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7693.93 MB 2025-02-14 18:28:39,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34835.79 MB 2025-02-14 18:28:39,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34835.79 MB 2025-02-14 18:28:39,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:28:39,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32384.15 MB 2025-02-14 18:28:39,301 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 18:28:39,301 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:28:39,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:28:39,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:28:39,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:28:39,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:28:39,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22184.70 MB 2025-02-14 18:28:39,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30602.85 MB 2025-02-14 18:28:39,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 18:28:39,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34835.79 MB 2025-02-14 18:28:39,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43205.53 MB 2025-02-14 18:28:39,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 18:28:39,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30602.85 MB 2025-02-14 18:28:39,464 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 18:28:39,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:28:39,466 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:28:39,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:28:39,467 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:28:39,471 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:28:39,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:28:39,472 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:28:39,472 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:30:01,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:01,155 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:30:01,160 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:30:01,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:01,164 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1357, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:30:01,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:01,166 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1357, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:30:21,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:30:21,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:30:21,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.81 seconds 2025-02-14 18:30:21,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:21,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22424.50 MB 2025-02-14 18:30:21,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27226.98 MB 2025-02-14 18:30:21,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4802.48 MB 2025-02-14 18:30:21,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55759.08 MB 2025-02-14 18:30:21,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35469.13 MB 2025-02-14 18:30:21,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20289.95 MB 2025-02-14 18:30:21,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36199.23 MB 2025-02-14 18:30:22,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:30:22,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:30:22,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 18:30:22,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:22,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27226.98 MB 2025-02-14 18:30:22,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21849.25 MB 2025-02-14 18:30:22,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5377.73 MB 2025-02-14 18:30:22,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35469.13 MB 2025-02-14 18:30:22,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35469.13 MB 2025-02-14 18:30:22,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:30:22,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30924.96 MB 2025-02-14 18:30:23,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:30:23,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:30:23,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.26 seconds 2025-02-14 18:30:23,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21849.25 MB 2025-02-14 18:30:23,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22194.29 MB 2025-02-14 18:30:23,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 345.05 MB 2025-02-14 18:30:23,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35469.13 MB 2025-02-14 18:30:23,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 18:30:23,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8986.30 MB 2025-02-14 18:30:23,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26189.80 MB 2025-02-14 18:30:23,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:30:23,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:30:23,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:30:23,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.29 MB 2025-02-14 18:30:23,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23422.27 MB 2025-02-14 18:30:23,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1227.97 MB 2025-02-14 18:30:23,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 18:30:23,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 18:30:23,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:30:23,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24343.60 MB 2025-02-14 18:30:23,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:30:23,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:30:23,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 18:30:23,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23422.27 MB 2025-02-14 18:30:23,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24879.50 MB 2025-02-14 18:30:23,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1457.23 MB 2025-02-14 18:30:23,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 18:30:23,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29863.44 MB 2025-02-14 18:30:23,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3380.61 MB 2025-02-14 18:30:23,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28487.45 MB 2025-02-14 18:30:23,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:30:23,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:30:23,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 18:30:23,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.29 MB 2025-02-14 18:30:23,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24879.50 MB 2025-02-14 18:30:23,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2685.20 MB 2025-02-14 18:30:23,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 18:30:23,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29863.44 MB 2025-02-14 18:30:23,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3380.61 MB 2025-02-14 18:30:23,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28487.45 MB 2025-02-14 18:30:23,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:30:23,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:30:23,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:30:23,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25876.30 MB 2025-02-14 18:30:23,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26374.85 MB 2025-02-14 18:30:23,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 498.55 MB 2025-02-14 18:30:23,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29863.44 MB 2025-02-14 18:30:23,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30131.88 MB 2025-02-14 18:30:23,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 18:30:23,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26834.91 MB 2025-02-14 18:30:23,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:30:23,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:30:23,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:30:23,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26643.23 MB 2025-02-14 18:30:23,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26856.83 MB 2025-02-14 18:30:23,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.60 MB 2025-02-14 18:30:23,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30131.88 MB 2025-02-14 18:30:23,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30131.88 MB 2025-02-14 18:30:23,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:30:23,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26931.78 MB 2025-02-14 18:30:23,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:30:23,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:30:23,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.40 seconds 2025-02-14 18:30:23,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17696.60 MB 2025-02-14 18:30:23,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27057.91 MB 2025-02-14 18:30:23,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9361.30 MB 2025-02-14 18:30:23,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55759.08 MB 2025-02-14 18:30:23,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30131.88 MB 2025-02-14 18:30:23,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25627.20 MB 2025-02-14 18:30:23,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27057.91 MB 2025-02-14 18:30:23,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:30:23,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:30:23,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:30:23,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27057.91 MB 2025-02-14 18:30:23,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30071.94 MB 2025-02-14 18:30:23,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 18:30:23,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30131.88 MB 2025-02-14 18:30:23,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31205.62 MB 2025-02-14 18:30:23,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1073.74 MB 2025-02-14 18:30:23,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30373.57 MB 2025-02-14 18:30:23,857 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:30:23,858 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:30:23,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:30:23,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:30:23,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:30:23,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:30:23,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22039.77 MB 2025-02-14 18:30:23,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.79 MB 2025-02-14 18:30:23,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:30:23,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31205.62 MB 2025-02-14 18:30:23,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39596.33 MB 2025-02-14 18:30:23,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:30:23,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30478.79 MB 2025-02-14 18:30:24,026 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:30:24,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:24,027 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:30:24,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:24,028 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:30:24,033 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:30:24,034 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:24,034 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:30:24,034 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:30:33,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:33,413 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:30:33,418 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:30:33,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:33,422 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1990, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:30:33,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:30:33,423 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1990, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:31:04,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:31:04,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:31:04,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.00 seconds 2025-02-14 18:31:04,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:04,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26835.35 MB 2025-02-14 18:31:04,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33877.84 MB 2025-02-14 18:31:04,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7042.50 MB 2025-02-14 18:31:04,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52181.34 MB 2025-02-14 18:31:04,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37721.47 MB 2025-02-14 18:31:04,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14459.86 MB 2025-02-14 18:31:04,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42875.80 MB 2025-02-14 18:31:04,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:31:04,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:31:04,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 18:31:04,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:04,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33877.84 MB 2025-02-14 18:31:04,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26124.29 MB 2025-02-14 18:31:04,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7753.55 MB 2025-02-14 18:31:04,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37721.47 MB 2025-02-14 18:31:04,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64091.06 MB 2025-02-14 18:31:04,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 26369.59 MB 2025-02-14 18:31:04,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53655.02 MB 2025-02-14 18:31:06,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:31:06,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:31:06,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 18:31:06,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:06,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26124.29 MB 2025-02-14 18:31:06,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26655.13 MB 2025-02-14 18:31:06,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:31:06,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64091.06 MB 2025-02-14 18:31:06,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32801.55 MB 2025-02-14 18:31:06,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31289.51 MB 2025-02-14 18:31:06,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30633.68 MB 2025-02-14 18:31:06,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:31:06,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:31:06,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:31:06,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:06,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26655.13 MB 2025-02-14 18:31:06,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28544.66 MB 2025-02-14 18:31:06,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:31:06,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32801.55 MB 2025-02-14 18:31:06,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32801.55 MB 2025-02-14 18:31:06,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:06,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29962.09 MB 2025-02-14 18:31:06,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:31:06,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:31:06,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:31:06,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:06,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28544.66 MB 2025-02-14 18:31:06,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30786.52 MB 2025-02-14 18:31:06,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:31:06,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32801.55 MB 2025-02-14 18:31:06,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38463.86 MB 2025-02-14 18:31:06,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:31:06,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36330.80 MB 2025-02-14 18:31:06,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:31:06,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:31:06,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:31:06,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:06,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26655.13 MB 2025-02-14 18:31:06,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30786.52 MB 2025-02-14 18:31:06,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:31:06,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32801.55 MB 2025-02-14 18:31:06,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38463.86 MB 2025-02-14 18:31:06,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:31:06,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36330.80 MB 2025-02-14 18:31:06,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:31:06,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:31:06,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:31:06,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:06,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32320.06 MB 2025-02-14 18:31:06,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33087.06 MB 2025-02-14 18:31:06,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:31:06,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38463.86 MB 2025-02-14 18:31:06,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38879.10 MB 2025-02-14 18:31:06,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:31:06,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33794.85 MB 2025-02-14 18:31:06,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:31:06,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:31:06,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:31:06,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:06,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33499.95 MB 2025-02-14 18:31:06,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33728.89 MB 2025-02-14 18:31:06,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-14 18:31:06,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38879.10 MB 2025-02-14 18:31:06,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38879.10 MB 2025-02-14 18:31:06,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:06,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33940.25 MB 2025-02-14 18:31:06,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:31:06,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:31:06,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.50 seconds 2025-02-14 18:31:06,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:06,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19902.03 MB 2025-02-14 18:31:06,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33929.74 MB 2025-02-14 18:31:06,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14027.72 MB 2025-02-14 18:31:06,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52181.34 MB 2025-02-14 18:31:06,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38879.10 MB 2025-02-14 18:31:06,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13302.24 MB 2025-02-14 18:31:06,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33940.25 MB 2025-02-14 18:31:07,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:31:07,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:31:07,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:31:07,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:07,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33929.74 MB 2025-02-14 18:31:07,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24902.99 MB 2025-02-14 18:31:07,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9026.75 MB 2025-02-14 18:31:07,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38879.10 MB 2025-02-14 18:31:07,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38879.10 MB 2025-02-14 18:31:07,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:07,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36438.64 MB 2025-02-14 18:31:07,213 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 18:31:07,213 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:31:07,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:31:07,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:31:07,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:31:07,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:07,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24902.99 MB 2025-02-14 18:31:07,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33333.39 MB 2025-02-14 18:31:07,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 18:31:07,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38879.10 MB 2025-02-14 18:31:07,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47259.32 MB 2025-02-14 18:31:07,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 18:31:07,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33333.39 MB 2025-02-14 18:31:07,378 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 18:31:07,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:07,380 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:31:07,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:07,381 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:31:07,385 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:31:07,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:07,387 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:31:07,387 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:31:16,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:16,440 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:31:16,445 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:31:16,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:16,449 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:31:16,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:16,450 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:31:19,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:31:19,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:31:19,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.22 seconds 2025-02-14 18:31:19,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:19,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14390.21 MB 2025-02-14 18:31:19,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15112.16 MB 2025-02-14 18:31:19,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-14 18:31:19,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55639.54 MB 2025-02-14 18:31:19,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-14 18:31:19,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35869.69 MB 2025-02-14 18:31:19,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.07 MB 2025-02-14 18:31:19,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:31:19,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:31:19,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:31:19,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:19,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15112.16 MB 2025-02-14 18:31:19,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15384.62 MB 2025-02-14 18:31:19,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.46 MB 2025-02-14 18:31:19,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-14 18:31:19,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-14 18:31:19,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:19,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17865.51 MB 2025-02-14 18:31:20,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:31:20,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:31:20,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.96 seconds 2025-02-14 18:31:20,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:20,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15384.62 MB 2025-02-14 18:31:20,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15640.75 MB 2025-02-14 18:31:20,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 18:31:20,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-14 18:31:20,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-14 18:31:20,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:20,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19640.24 MB 2025-02-14 18:31:20,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:31:20,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:31:20,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:31:20,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:20,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-14 18:31:20,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16552.23 MB 2025-02-14 18:31:20,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 18:31:20,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-14 18:31:20,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-14 18:31:20,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:20,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17236.14 MB 2025-02-14 18:31:20,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:31:20,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:31:20,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:31:20,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:20,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16552.23 MB 2025-02-14 18:31:20,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-14 18:31:20,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-14 18:31:20,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-14 18:31:20,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21827.16 MB 2025-02-14 18:31:20,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2057.31 MB 2025-02-14 18:31:20,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20311.79 MB 2025-02-14 18:31:20,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:31:20,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:31:20,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:31:20,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:20,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-14 18:31:20,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-14 18:31:20,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-14 18:31:20,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-14 18:31:20,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21827.16 MB 2025-02-14 18:31:20,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2057.31 MB 2025-02-14 18:31:20,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20311.79 MB 2025-02-14 18:31:20,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:31:20,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:31:20,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:31:20,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:20,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18373.89 MB 2025-02-14 18:31:20,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18745.80 MB 2025-02-14 18:31:20,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.91 MB 2025-02-14 18:31:20,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21827.16 MB 2025-02-14 18:31:20,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22022.19 MB 2025-02-14 18:31:20,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-14 18:31:20,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19090.99 MB 2025-02-14 18:31:20,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:31:20,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:31:20,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:31:20,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:20,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18945.03 MB 2025-02-14 18:31:20,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19172.07 MB 2025-02-14 18:31:20,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.04 MB 2025-02-14 18:31:20,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22022.19 MB 2025-02-14 18:31:20,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22024.29 MB 2025-02-14 18:31:20,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 18:31:20,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19207.79 MB 2025-02-14 18:31:20,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:31:20,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:31:20,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.41 seconds 2025-02-14 18:31:20,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:20,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13679.46 MB 2025-02-14 18:31:20,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19373.15 MB 2025-02-14 18:31:20,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5693.69 MB 2025-02-14 18:31:20,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55639.54 MB 2025-02-14 18:31:20,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22024.29 MB 2025-02-14 18:31:20,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33615.25 MB 2025-02-14 18:31:20,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19373.15 MB 2025-02-14 18:31:21,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:31:21,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:31:21,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:31:21,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:21,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19373.15 MB 2025-02-14 18:31:21,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17708.79 MB 2025-02-14 18:31:21,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1664.36 MB 2025-02-14 18:31:21,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22024.29 MB 2025-02-14 18:31:21,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22024.29 MB 2025-02-14 18:31:21,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:21,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19373.15 MB 2025-02-14 18:31:21,147 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:31:21,147 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:31:21,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:31:21,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:31:21,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:31:21,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:21,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17708.79 MB 2025-02-14 18:31:21,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26147.81 MB 2025-02-14 18:31:21,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:31:21,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22024.29 MB 2025-02-14 18:31:21,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30415.00 MB 2025-02-14 18:31:21,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:31:21,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26147.81 MB 2025-02-14 18:31:21,313 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:31:21,314 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:21,314 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:31:21,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:21,315 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:31:21,320 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:31:21,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:21,321 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:31:21,321 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:31:30,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:30,528 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:31:30,536 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:31:30,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:30,542 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:31:30,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:30,544 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:31:33,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:31:33,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:31:33,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.77 seconds 2025-02-14 18:31:33,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:33,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-14 18:31:33,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-14 18:31:33,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-14 18:31:33,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43000.00 MB 2025-02-14 18:31:33,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22313.70 MB 2025-02-14 18:31:33,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20686.31 MB 2025-02-14 18:31:33,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23652.54 MB 2025-02-14 18:31:33,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:31:33,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:31:33,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:31:33,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:33,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-14 18:31:33,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15003.92 MB 2025-02-14 18:31:33,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.98 MB 2025-02-14 18:31:33,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22313.70 MB 2025-02-14 18:31:33,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22313.70 MB 2025-02-14 18:31:33,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:33,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17107.90 MB 2025-02-14 18:31:34,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:31:34,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:31:34,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 18:31:34,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15003.92 MB 2025-02-14 18:31:34,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15217.58 MB 2025-02-14 18:31:34,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 18:31:34,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22313.70 MB 2025-02-14 18:31:34,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21856.52 MB 2025-02-14 18:31:34,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -457.18 MB 2025-02-14 18:31:34,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19173.57 MB 2025-02-14 18:31:34,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:31:34,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:31:34,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:31:34,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-14 18:31:34,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15977.94 MB 2025-02-14 18:31:34,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 18:31:34,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21856.52 MB 2025-02-14 18:31:34,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21856.52 MB 2025-02-14 18:31:34,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:34,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16548.46 MB 2025-02-14 18:31:34,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:31:34,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:31:34,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:31:34,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.94 MB 2025-02-14 18:31:34,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.32 MB 2025-02-14 18:31:34,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 18:31:34,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21856.52 MB 2025-02-14 18:31:34,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21856.52 MB 2025-02-14 18:31:34,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:34,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19111.86 MB 2025-02-14 18:31:34,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:31:34,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:31:34,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 18:31:34,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-14 18:31:34,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.32 MB 2025-02-14 18:31:34,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 18:31:34,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21856.52 MB 2025-02-14 18:31:34,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21856.52 MB 2025-02-14 18:31:34,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:34,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19111.86 MB 2025-02-14 18:31:34,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:31:34,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:31:34,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:31:34,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17497.57 MB 2025-02-14 18:31:34,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17806.29 MB 2025-02-14 18:31:34,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 18:31:34,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21856.52 MB 2025-02-14 18:31:34,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22020.10 MB 2025-02-14 18:31:34,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 18:31:34,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18101.26 MB 2025-02-14 18:31:34,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:31:34,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:31:34,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:31:34,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17972.49 MB 2025-02-14 18:31:34,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18199.95 MB 2025-02-14 18:31:34,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.47 MB 2025-02-14 18:31:34,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22020.10 MB 2025-02-14 18:31:34,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22020.10 MB 2025-02-14 18:31:34,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:34,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18222.03 MB 2025-02-14 18:31:34,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:31:34,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:31:34,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.84 seconds 2025-02-14 18:31:34,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-14 18:31:34,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18401.03 MB 2025-02-14 18:31:34,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4826.09 MB 2025-02-14 18:31:34,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43000.00 MB 2025-02-14 18:31:34,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22020.10 MB 2025-02-14 18:31:34,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20979.91 MB 2025-02-14 18:31:34,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18401.03 MB 2025-02-14 18:31:34,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:31:34,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:31:34,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 18:31:34,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18401.03 MB 2025-02-14 18:31:34,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17451.41 MB 2025-02-14 18:31:34,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -949.61 MB 2025-02-14 18:31:34,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22020.10 MB 2025-02-14 18:31:34,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22020.10 MB 2025-02-14 18:31:34,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:31:34,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19204.76 MB 2025-02-14 18:31:34,699 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:31:34,699 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:31:34,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:31:34,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:31:34,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:31:34,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:31:34,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17451.41 MB 2025-02-14 18:31:34,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25890.44 MB 2025-02-14 18:31:34,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:31:34,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22020.10 MB 2025-02-14 18:31:34,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30410.80 MB 2025-02-14 18:31:34,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:31:34,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25890.44 MB 2025-02-14 18:31:34,954 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:31:34,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:34,957 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:31:34,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:34,959 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:31:34,966 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:31:34,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:31:34,968 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:31:34,968 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:32:08,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:08,835 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:32:08,840 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:32:08,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:08,843 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:32:08,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:08,844 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:32:11,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:32:11,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:32:11,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.37 seconds 2025-02-14 18:32:11,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:11,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-14 18:32:11,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-14 18:32:11,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 18:32:11,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42995.81 MB 2025-02-14 18:32:11,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21854.42 MB 2025-02-14 18:32:11,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21141.39 MB 2025-02-14 18:32:11,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23506.21 MB 2025-02-14 18:32:11,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:32:11,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:32:11,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:32:11,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:11,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-14 18:32:11,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.49 MB 2025-02-14 18:32:11,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.20 MB 2025-02-14 18:32:11,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21854.42 MB 2025-02-14 18:32:11,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21854.42 MB 2025-02-14 18:32:11,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:11,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16672.98 MB 2025-02-14 18:32:11,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:32:11,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:32:11,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 18:32:11,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:11,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.49 MB 2025-02-14 18:32:11,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14991.58 MB 2025-02-14 18:32:11,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-14 18:32:11,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21854.42 MB 2025-02-14 18:32:11,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21397.24 MB 2025-02-14 18:32:11,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -457.18 MB 2025-02-14 18:32:11,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18966.14 MB 2025-02-14 18:32:11,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:32:11,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:32:11,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:32:11,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:11,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14991.51 MB 2025-02-14 18:32:11,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15685.75 MB 2025-02-14 18:32:11,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-14 18:32:11,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21397.24 MB 2025-02-14 18:32:11,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21397.24 MB 2025-02-14 18:32:11,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:11,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16206.66 MB 2025-02-14 18:32:12,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:32:12,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:32:12,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:32:12,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:12,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15685.75 MB 2025-02-14 18:32:12,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16509.67 MB 2025-02-14 18:32:12,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-14 18:32:12,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21397.24 MB 2025-02-14 18:32:12,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21397.24 MB 2025-02-14 18:32:12,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:12,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18547.15 MB 2025-02-14 18:32:12,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:32:12,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:32:12,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:32:12,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:12,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14991.51 MB 2025-02-14 18:32:12,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16509.67 MB 2025-02-14 18:32:12,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-14 18:32:12,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21397.24 MB 2025-02-14 18:32:12,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21397.24 MB 2025-02-14 18:32:12,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:12,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18547.15 MB 2025-02-14 18:32:12,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:32:12,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:32:12,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:32:12,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:12,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17073.25 MB 2025-02-14 18:32:12,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17355.12 MB 2025-02-14 18:32:12,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.87 MB 2025-02-14 18:32:12,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21397.24 MB 2025-02-14 18:32:12,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21548.24 MB 2025-02-14 18:32:12,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 18:32:12,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17625.26 MB 2025-02-14 18:32:12,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:32:12,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:32:12,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:32:12,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:12,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17506.86 MB 2025-02-14 18:32:12,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17715.05 MB 2025-02-14 18:32:12,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.19 MB 2025-02-14 18:32:12,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21548.24 MB 2025-02-14 18:32:12,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21550.33 MB 2025-02-14 18:32:12,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 18:32:12,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17720.43 MB 2025-02-14 18:32:12,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:32:12,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:32:12,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.27 seconds 2025-02-14 18:32:12,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:12,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-14 18:32:12,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17915.70 MB 2025-02-14 18:32:12,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4413.93 MB 2025-02-14 18:32:12,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42995.81 MB 2025-02-14 18:32:12,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21550.33 MB 2025-02-14 18:32:12,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21445.48 MB 2025-02-14 18:32:12,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17915.70 MB 2025-02-14 18:32:12,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:32:12,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:32:12,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:32:12,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:12,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17915.70 MB 2025-02-14 18:32:12,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17305.70 MB 2025-02-14 18:32:12,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -610.00 MB 2025-02-14 18:32:12,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21550.33 MB 2025-02-14 18:32:12,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21550.33 MB 2025-02-14 18:32:12,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:12,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19018.54 MB 2025-02-14 18:32:12,401 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 18:32:12,401 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:32:12,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:32:12,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:32:12,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:32:12,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:12,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17305.70 MB 2025-02-14 18:32:12,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25727.66 MB 2025-02-14 18:32:12,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 18:32:12,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21550.33 MB 2025-02-14 18:32:12,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29922.16 MB 2025-02-14 18:32:12,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 18:32:12,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25727.66 MB 2025-02-14 18:32:12,566 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 18:32:12,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:12,568 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:32:12,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:12,569 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:32:12,573 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:32:12,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:12,574 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:32:12,574 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:32:23,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:23,051 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:32:23,056 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:32:23,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:23,059 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 692, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:32:23,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:23,060 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 692, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:32:33,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:32:33,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:32:33,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.69 seconds 2025-02-14 18:32:33,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:33,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17790.67 MB 2025-02-14 18:32:33,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20239.62 MB 2025-02-14 18:32:33,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2448.95 MB 2025-02-14 18:32:33,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38294.00 MB 2025-02-14 18:32:33,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24652.02 MB 2025-02-14 18:32:33,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13641.97 MB 2025-02-14 18:32:33,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29073.98 MB 2025-02-14 18:32:33,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:32:33,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:32:33,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 18:32:33,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:33,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20239.62 MB 2025-02-14 18:32:33,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19376.39 MB 2025-02-14 18:32:33,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -863.23 MB 2025-02-14 18:32:33,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24652.02 MB 2025-02-14 18:32:33,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32346.47 MB 2025-02-14 18:32:33,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7694.45 MB 2025-02-14 18:32:33,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28630.94 MB 2025-02-14 18:32:35,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:32:35,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:32:35,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 18:32:35,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:35,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19376.39 MB 2025-02-14 18:32:35,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19907.23 MB 2025-02-14 18:32:35,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:32:35,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32346.47 MB 2025-02-14 18:32:35,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26776.44 MB 2025-02-14 18:32:35,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5570.04 MB 2025-02-14 18:32:35,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23885.78 MB 2025-02-14 18:32:35,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:32:35,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:32:35,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:32:35,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:35,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19907.23 MB 2025-02-14 18:32:35,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21796.77 MB 2025-02-14 18:32:35,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:32:35,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26776.44 MB 2025-02-14 18:32:35,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26776.44 MB 2025-02-14 18:32:35,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:35,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23214.20 MB 2025-02-14 18:32:35,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:32:35,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:32:35,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:32:35,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:35,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21796.77 MB 2025-02-14 18:32:35,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24038.62 MB 2025-02-14 18:32:35,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:32:35,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26776.44 MB 2025-02-14 18:32:35,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31968.99 MB 2025-02-14 18:32:35,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-14 18:32:35,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29582.90 MB 2025-02-14 18:32:35,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:32:35,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:32:35,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:32:35,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:35,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19907.23 MB 2025-02-14 18:32:35,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24038.62 MB 2025-02-14 18:32:35,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:32:35,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26776.44 MB 2025-02-14 18:32:35,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31968.99 MB 2025-02-14 18:32:35,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-14 18:32:35,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29582.90 MB 2025-02-14 18:32:36,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:32:36,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:32:36,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:32:36,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:36,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25572.17 MB 2025-02-14 18:32:36,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26339.17 MB 2025-02-14 18:32:36,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:32:36,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31968.99 MB 2025-02-14 18:32:36,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-14 18:32:36,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:32:36,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27046.96 MB 2025-02-14 18:32:36,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:32:36,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:32:36,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:32:36,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:36,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26752.06 MB 2025-02-14 18:32:36,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26980.22 MB 2025-02-14 18:32:36,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.16 MB 2025-02-14 18:32:36,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-14 18:32:36,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-14 18:32:36,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:36,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27170.94 MB 2025-02-14 18:32:36,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:32:36,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:32:36,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.06 seconds 2025-02-14 18:32:36,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:36,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15379.69 MB 2025-02-14 18:32:36,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27180.70 MB 2025-02-14 18:32:36,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11801.01 MB 2025-02-14 18:32:36,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38294.00 MB 2025-02-14 18:32:36,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-14 18:32:36,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5907.68 MB 2025-02-14 18:32:36,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27180.70 MB 2025-02-14 18:32:36,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:32:36,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:32:36,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:32:36,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:36,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27180.70 MB 2025-02-14 18:32:36,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20372.09 MB 2025-02-14 18:32:36,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6808.61 MB 2025-02-14 18:32:36,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-14 18:32:36,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32386.32 MB 2025-02-14 18:32:36,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:36,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29682.54 MB 2025-02-14 18:32:36,409 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-14 18:32:36,410 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:32:36,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:32:36,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:32:36,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:32:36,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:36,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20372.09 MB 2025-02-14 18:32:36,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28777.75 MB 2025-02-14 18:32:36,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-14 18:32:36,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32386.32 MB 2025-02-14 18:32:36,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36565.94 MB 2025-02-14 18:32:36,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 18:32:36,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28777.75 MB 2025-02-14 18:32:36,574 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-14 18:32:36,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:36,575 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:32:36,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:36,576 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:32:36,581 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:32:36,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:36,582 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:32:36,582 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:32:43,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:43,739 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:32:43,744 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:32:43,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:43,747 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 228, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:32:43,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:43,748 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 228, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:32:47,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:32:47,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:32:47,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.54 seconds 2025-02-14 18:32:47,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:47,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14557.45 MB 2025-02-14 18:32:47,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15364.33 MB 2025-02-14 18:32:47,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 806.88 MB 2025-02-14 18:32:47,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44925.19 MB 2025-02-14 18:32:47,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22802.33 MB 2025-02-14 18:32:47,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22122.86 MB 2025-02-14 18:32:47,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24255.31 MB 2025-02-14 18:32:47,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:32:47,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:32:47,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:32:47,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:47,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15364.33 MB 2025-02-14 18:32:47,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15389.99 MB 2025-02-14 18:32:47,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.67 MB 2025-02-14 18:32:47,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22802.33 MB 2025-02-14 18:32:47,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22802.33 MB 2025-02-14 18:32:47,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:47,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17850.59 MB 2025-02-14 18:32:48,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:32:48,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:32:48,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 18:32:48,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15389.99 MB 2025-02-14 18:32:48,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15623.56 MB 2025-02-14 18:32:48,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.57 MB 2025-02-14 18:32:48,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22802.33 MB 2025-02-14 18:32:48,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22802.33 MB 2025-02-14 18:32:48,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:48,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19559.64 MB 2025-02-14 18:32:48,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:32:48,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:32:48,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:32:48,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15623.56 MB 2025-02-14 18:32:48,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16454.76 MB 2025-02-14 18:32:48,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.19 MB 2025-02-14 18:32:48,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22802.33 MB 2025-02-14 18:32:48,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 18:32:48,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 18:32:48,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17078.43 MB 2025-02-14 18:32:48,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:32:48,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:32:48,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:32:48,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16454.76 MB 2025-02-14 18:32:48,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17441.21 MB 2025-02-14 18:32:48,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 986.45 MB 2025-02-14 18:32:48,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 18:32:48,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 18:32:48,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:48,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.66 MB 2025-02-14 18:32:48,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:32:48,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:32:48,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:32:48,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15623.56 MB 2025-02-14 18:32:48,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17441.21 MB 2025-02-14 18:32:48,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1817.65 MB 2025-02-14 18:32:48,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22802.33 MB 2025-02-14 18:32:48,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 18:32:48,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 18:32:48,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.66 MB 2025-02-14 18:32:48,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:32:48,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:32:48,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:32:48,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18115.97 MB 2025-02-14 18:32:48,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18453.45 MB 2025-02-14 18:32:48,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.48 MB 2025-02-14 18:32:48,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 18:32:48,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 18:32:48,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 18:32:48,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18770.19 MB 2025-02-14 18:32:48,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:32:48,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:32:48,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:32:48,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18635.13 MB 2025-02-14 18:32:48,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18862.45 MB 2025-02-14 18:32:48,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.32 MB 2025-02-14 18:32:48,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-14 18:32:48,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 18:32:48,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:48,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18886.61 MB 2025-02-14 18:32:48,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:32:48,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:32:48,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.60 seconds 2025-02-14 18:32:48,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13763.08 MB 2025-02-14 18:32:48,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19063.15 MB 2025-02-14 18:32:48,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5300.08 MB 2025-02-14 18:32:48,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44925.19 MB 2025-02-14 18:32:48,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 18:32:48,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21938.31 MB 2025-02-14 18:32:48,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19063.15 MB 2025-02-14 18:32:48,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:32:48,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:32:48,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:32:48,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19063.15 MB 2025-02-14 18:32:48,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17704.63 MB 2025-02-14 18:32:48,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1358.52 MB 2025-02-14 18:32:48,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-14 18:32:48,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 18:32:48,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:32:48,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19297.15 MB 2025-02-14 18:32:48,638 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 18:32:48,638 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:32:48,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:32:48,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:32:48,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:32:48,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:32:48,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17704.63 MB 2025-02-14 18:32:48,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26127.84 MB 2025-02-14 18:32:48,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 18:32:48,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-14 18:32:48,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31362.91 MB 2025-02-14 18:32:48,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 18:32:48,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26127.84 MB 2025-02-14 18:32:48,802 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 18:32:48,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:48,803 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:32:48,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:48,804 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:32:48,809 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:32:48,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:32:48,810 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:32:48,810 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:34:14,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:14,521 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:34:14,529 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:34:14,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:14,536 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:34:14,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:14,538 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:34:16,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:34:16,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:34:16,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.38 seconds 2025-02-14 18:34:16,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:16,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-14 18:34:16,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-14 18:34:16,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-14 18:34:16,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39738.93 MB 2025-02-14 18:34:16,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 18:34:16,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16932.41 MB 2025-02-14 18:34:16,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23499.24 MB 2025-02-14 18:34:16,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:34:16,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:34:16,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:34:16,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:16,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-14 18:34:16,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14784.27 MB 2025-02-14 18:34:16,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.48 MB 2025-02-14 18:34:16,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 18:34:16,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 18:34:16,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:34:16,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.89 MB 2025-02-14 18:34:17,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:34:17,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:34:17,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 18:34:17,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:17,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.27 MB 2025-02-14 18:34:17,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14978.03 MB 2025-02-14 18:34:17,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 193.76 MB 2025-02-14 18:34:17,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 18:34:17,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 18:34:17,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:34:17,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18953.92 MB 2025-02-14 18:34:17,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:34:17,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:34:17,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:34:17,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:17,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-14 18:34:17,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15667.47 MB 2025-02-14 18:34:17,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 689.51 MB 2025-02-14 18:34:17,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 18:34:17,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 18:34:17,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:34:17,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16184.84 MB 2025-02-14 18:34:17,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:34:17,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:34:17,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:34:17,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:17,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15667.47 MB 2025-02-14 18:34:17,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-14 18:34:17,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 818.32 MB 2025-02-14 18:34:17,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 18:34:17,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 18:34:17,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:34:17,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-14 18:34:17,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:34:17,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:34:17,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:34:17,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:17,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-14 18:34:17,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-14 18:34:17,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1507.83 MB 2025-02-14 18:34:17,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 18:34:17,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 18:34:17,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:34:17,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-14 18:34:17,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:34:17,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:34:17,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:34:17,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:17,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17045.54 MB 2025-02-14 18:34:17,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17325.49 MB 2025-02-14 18:34:17,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.96 MB 2025-02-14 18:34:17,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 18:34:17,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22955.43 MB 2025-02-14 18:34:17,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 18:34:17,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17594.33 MB 2025-02-14 18:34:17,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:34:17,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:34:17,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:34:17,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:17,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17476.20 MB 2025-02-14 18:34:17,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17681.39 MB 2025-02-14 18:34:17,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.19 MB 2025-02-14 18:34:17,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22955.43 MB 2025-02-14 18:34:17,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22959.62 MB 2025-02-14 18:34:17,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 18:34:17,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17696.37 MB 2025-02-14 18:34:17,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:34:17,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:34:17,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.29 seconds 2025-02-14 18:34:17,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:17,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-14 18:34:17,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17882.24 MB 2025-02-14 18:34:17,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4383.95 MB 2025-02-14 18:34:17,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39738.93 MB 2025-02-14 18:34:17,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22959.62 MB 2025-02-14 18:34:17,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16779.31 MB 2025-02-14 18:34:17,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17882.24 MB 2025-02-14 18:34:18,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:34:18,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:34:18,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:34:18,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:18,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17882.24 MB 2025-02-14 18:34:18,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17300.55 MB 2025-02-14 18:34:18,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -581.69 MB 2025-02-14 18:34:18,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22959.62 MB 2025-02-14 18:34:18,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22959.62 MB 2025-02-14 18:34:18,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:34:18,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18986.16 MB 2025-02-14 18:34:18,114 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 18:34:18,115 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:34:18,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:34:18,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:34:18,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:34:18,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:34:18,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17300.55 MB 2025-02-14 18:34:18,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25730.95 MB 2025-02-14 18:34:18,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 18:34:18,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22959.62 MB 2025-02-14 18:34:18,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31339.84 MB 2025-02-14 18:34:18,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 18:34:18,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25730.95 MB 2025-02-14 18:34:18,281 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 18:34:18,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:18,282 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:34:18,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:18,283 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:34:18,288 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:34:18,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:18,289 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:34:18,289 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:34:52,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:52,770 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:34:52,775 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:34:52,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:52,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1933, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:34:52,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:34:52,780 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1933, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:35:22,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:35:22,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:35:22,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.72 seconds 2025-02-14 18:35:22,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:22,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26438.16 MB 2025-02-14 18:35:22,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33279.07 MB 2025-02-14 18:35:22,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6840.91 MB 2025-02-14 18:35:22,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39720.06 MB 2025-02-14 18:35:22,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37526.44 MB 2025-02-14 18:35:22,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2193.62 MB 2025-02-14 18:35:22,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42252.12 MB 2025-02-14 18:35:22,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:35:22,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:35:22,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 18:35:22,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:22,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33279.07 MB 2025-02-14 18:35:22,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25826.91 MB 2025-02-14 18:35:22,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7452.15 MB 2025-02-14 18:35:22,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37526.44 MB 2025-02-14 18:35:22,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62249.76 MB 2025-02-14 18:35:22,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24723.32 MB 2025-02-14 18:35:22,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52645.58 MB 2025-02-14 18:35:24,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:35:24,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:35:24,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 18:35:24,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:24,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25826.91 MB 2025-02-14 18:35:24,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26357.76 MB 2025-02-14 18:35:24,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:35:24,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62249.76 MB 2025-02-14 18:35:24,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 18:35:24,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30148.66 MB 2025-02-14 18:35:24,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.30 MB 2025-02-14 18:35:24,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:35:24,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:35:24,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:35:24,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:24,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26357.76 MB 2025-02-14 18:35:24,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28247.29 MB 2025-02-14 18:35:24,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:35:24,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 18:35:24,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 18:35:24,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:35:24,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29664.72 MB 2025-02-14 18:35:24,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:35:24,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:35:24,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:35:24,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:24,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28247.29 MB 2025-02-14 18:35:24,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30489.15 MB 2025-02-14 18:35:24,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:35:24,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 18:35:24,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38235.28 MB 2025-02-14 18:35:24,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:35:24,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36033.43 MB 2025-02-14 18:35:24,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:35:24,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:35:24,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:35:24,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:24,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26357.76 MB 2025-02-14 18:35:24,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30489.15 MB 2025-02-14 18:35:24,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:35:24,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 18:35:24,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38235.28 MB 2025-02-14 18:35:24,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:35:24,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36033.43 MB 2025-02-14 18:35:24,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:35:24,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:35:24,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:35:24,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:24,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32022.69 MB 2025-02-14 18:35:24,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32789.69 MB 2025-02-14 18:35:24,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:35:24,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38235.28 MB 2025-02-14 18:35:24,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38650.51 MB 2025-02-14 18:35:24,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:35:24,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33497.48 MB 2025-02-14 18:35:24,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:35:24,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:35:24,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:35:24,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:24,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33202.58 MB 2025-02-14 18:35:24,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33429.89 MB 2025-02-14 18:35:24,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.31 MB 2025-02-14 18:35:24,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38650.51 MB 2025-02-14 18:35:24,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38650.51 MB 2025-02-14 18:35:24,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:35:24,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33639.62 MB 2025-02-14 18:35:24,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:35:24,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:35:24,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.19 seconds 2025-02-14 18:35:24,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:24,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19703.43 MB 2025-02-14 18:35:24,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33630.38 MB 2025-02-14 18:35:24,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13926.94 MB 2025-02-14 18:35:24,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39720.06 MB 2025-02-14 18:35:24,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38650.51 MB 2025-02-14 18:35:24,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1069.55 MB 2025-02-14 18:35:24,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33639.62 MB 2025-02-14 18:35:25,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:35:25,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:35:25,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:35:25,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:25,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33630.38 MB 2025-02-14 18:35:25,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24697.25 MB 2025-02-14 18:35:25,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8933.12 MB 2025-02-14 18:35:25,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38650.51 MB 2025-02-14 18:35:25,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38650.51 MB 2025-02-14 18:35:25,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:35:25,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36133.44 MB 2025-02-14 18:35:25,264 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 18:35:25,264 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:35:25,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:35:25,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:35:25,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:35:25,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:35:25,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24697.25 MB 2025-02-14 18:35:25,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33107.06 MB 2025-02-14 18:35:25,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 18:35:25,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38650.51 MB 2025-02-14 18:35:25,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42830.14 MB 2025-02-14 18:35:25,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 18:35:25,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33107.06 MB 2025-02-14 18:35:25,430 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 18:35:25,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:35:25,432 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:35:25,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:35:25,433 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:35:25,437 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:35:25,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:35:25,438 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:35:25,439 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:36:24,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:36:24,120 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:36:24,125 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:36:24,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:36:24,128 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 822, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:36:24,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:36:24,129 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 822, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:36:36,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:36:36,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:36:36,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.58 seconds 2025-02-14 18:36:36,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:36,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18696.53 MB 2025-02-14 18:36:36,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21605.55 MB 2025-02-14 18:36:36,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2909.01 MB 2025-02-14 18:36:36,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55371.10 MB 2025-02-14 18:36:36,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26621.25 MB 2025-02-14 18:36:36,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28749.86 MB 2025-02-14 18:36:36,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30432.83 MB 2025-02-14 18:36:36,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:36:36,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:36:36,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 18:36:36,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:36,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21605.55 MB 2025-02-14 18:36:36,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20052.22 MB 2025-02-14 18:36:36,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1553.33 MB 2025-02-14 18:36:36,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26621.25 MB 2025-02-14 18:36:36,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34990.98 MB 2025-02-14 18:36:36,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 18:36:36,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31359.74 MB 2025-02-14 18:36:38,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:36:38,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:36:38,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:36:38,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:38,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20052.22 MB 2025-02-14 18:36:38,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20583.06 MB 2025-02-14 18:36:38,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:36:38,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34990.98 MB 2025-02-14 18:36:38,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25834.82 MB 2025-02-14 18:36:38,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9156.17 MB 2025-02-14 18:36:38,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24561.61 MB 2025-02-14 18:36:38,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:36:38,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:36:38,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:36:38,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:38,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20583.06 MB 2025-02-14 18:36:38,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22472.60 MB 2025-02-14 18:36:38,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:36:38,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25834.82 MB 2025-02-14 18:36:38,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25834.82 MB 2025-02-14 18:36:38,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:36:38,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23890.03 MB 2025-02-14 18:36:38,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:36:38,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:36:38,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:36:38,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:38,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22472.60 MB 2025-02-14 18:36:38,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24714.45 MB 2025-02-14 18:36:38,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:36:38,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25834.82 MB 2025-02-14 18:36:38,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31968.99 MB 2025-02-14 18:36:38,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:36:38,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30258.73 MB 2025-02-14 18:36:38,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:36:38,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:36:38,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:36:38,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:38,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20583.06 MB 2025-02-14 18:36:38,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24714.45 MB 2025-02-14 18:36:38,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:36:38,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25834.82 MB 2025-02-14 18:36:38,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31968.99 MB 2025-02-14 18:36:38,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:36:38,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30258.73 MB 2025-02-14 18:36:39,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:36:39,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:36:39,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 18:36:39,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:39,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26247.99 MB 2025-02-14 18:36:39,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27015.00 MB 2025-02-14 18:36:39,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:36:39,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31968.99 MB 2025-02-14 18:36:39,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32384.22 MB 2025-02-14 18:36:39,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:36:39,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27722.79 MB 2025-02-14 18:36:39,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:36:39,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:36:39,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:36:39,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:39,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27427.89 MB 2025-02-14 18:36:39,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27655.10 MB 2025-02-14 18:36:39,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.22 MB 2025-02-14 18:36:39,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32384.22 MB 2025-02-14 18:36:39,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32384.22 MB 2025-02-14 18:36:39,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:36:39,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27849.97 MB 2025-02-14 18:36:39,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:36:39,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:36:39,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.97 seconds 2025-02-14 18:36:39,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:39,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15832.62 MB 2025-02-14 18:36:39,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27856.17 MB 2025-02-14 18:36:39,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12023.55 MB 2025-02-14 18:36:39,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55371.10 MB 2025-02-14 18:36:39,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32384.22 MB 2025-02-14 18:36:39,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22986.88 MB 2025-02-14 18:36:39,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27856.17 MB 2025-02-14 18:36:39,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:36:39,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:36:39,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:36:39,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:39,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27856.17 MB 2025-02-14 18:36:39,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20837.01 MB 2025-02-14 18:36:39,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7019.17 MB 2025-02-14 18:36:39,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32384.22 MB 2025-02-14 18:36:39,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32384.22 MB 2025-02-14 18:36:39,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:36:39,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30367.84 MB 2025-02-14 18:36:39,385 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:36:39,385 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:36:39,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:36:39,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:36:39,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:36:39,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:36:39,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20837.01 MB 2025-02-14 18:36:39,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29276.03 MB 2025-02-14 18:36:39,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:36:39,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32384.22 MB 2025-02-14 18:36:39,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40774.93 MB 2025-02-14 18:36:39,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:36:39,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29276.03 MB 2025-02-14 18:36:39,552 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:36:39,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:36:39,554 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:36:39,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:36:39,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:36:39,560 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:36:39,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:36:39,561 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:36:39,561 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:38:04,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:38:04,466 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:38:04,471 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:38:04,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:38:04,474 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1669, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:38:04,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:38:04,475 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1669, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:38:30,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:38:30,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:38:30,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.56 seconds 2025-02-14 18:38:30,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:30,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24598.57 MB 2025-02-14 18:38:30,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30505.06 MB 2025-02-14 18:38:30,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5906.50 MB 2025-02-14 18:38:30,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53359.94 MB 2025-02-14 18:38:30,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36591.11 MB 2025-02-14 18:38:30,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16768.83 MB 2025-02-14 18:38:30,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39507.54 MB 2025-02-14 18:38:30,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:38:30,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:38:30,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:38:30,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:30,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30505.06 MB 2025-02-14 18:38:30,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24454.46 MB 2025-02-14 18:38:30,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6050.60 MB 2025-02-14 18:38:30,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36591.11 MB 2025-02-14 18:38:30,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44480.59 MB 2025-02-14 18:38:30,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7889.49 MB 2025-02-14 18:38:30,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39805.87 MB 2025-02-14 18:38:32,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:38:32,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:38:32,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 18:38:32,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24454.46 MB 2025-02-14 18:38:32,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24985.30 MB 2025-02-14 18:38:32,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:38:32,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44480.59 MB 2025-02-14 18:38:32,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27919.38 MB 2025-02-14 18:38:32,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16561.21 MB 2025-02-14 18:38:32,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28964.89 MB 2025-02-14 18:38:32,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:38:32,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:38:32,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:38:32,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24985.30 MB 2025-02-14 18:38:32,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26874.84 MB 2025-02-14 18:38:32,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:38:32,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27919.38 MB 2025-02-14 18:38:32,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29806.82 MB 2025-02-14 18:38:32,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 18:38:32,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28292.27 MB 2025-02-14 18:38:32,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:38:32,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:38:32,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:38:32,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26874.84 MB 2025-02-14 18:38:32,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29116.69 MB 2025-02-14 18:38:32,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:38:32,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29806.82 MB 2025-02-14 18:38:32,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-14 18:38:32,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 18:38:32,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34660.97 MB 2025-02-14 18:38:32,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:38:32,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:38:32,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 18:38:32,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24985.30 MB 2025-02-14 18:38:32,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29116.69 MB 2025-02-14 18:38:32,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:38:32,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27919.38 MB 2025-02-14 18:38:32,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-14 18:38:32,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 18:38:32,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34660.97 MB 2025-02-14 18:38:32,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:38:32,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:38:32,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:38:32,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30650.24 MB 2025-02-14 18:38:32,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31417.24 MB 2025-02-14 18:38:32,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:38:32,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36412.85 MB 2025-02-14 18:38:32,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 18:38:32,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:38:32,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32125.03 MB 2025-02-14 18:38:32,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:38:32,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:38:32,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:38:32,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31830.13 MB 2025-02-14 18:38:32,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32060.06 MB 2025-02-14 18:38:32,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.93 MB 2025-02-14 18:38:32,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36828.09 MB 2025-02-14 18:38:32,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 18:38:32,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:38:32,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32251.41 MB 2025-02-14 18:38:32,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:38:32,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:38:32,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.99 seconds 2025-02-14 18:38:32,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18783.64 MB 2025-02-14 18:38:32,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32261.13 MB 2025-02-14 18:38:32,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13477.49 MB 2025-02-14 18:38:32,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53359.94 MB 2025-02-14 18:38:32,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 18:38:32,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16531.85 MB 2025-02-14 18:38:32,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32261.13 MB 2025-02-14 18:38:32,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:38:32,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:38:32,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:38:32,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32261.13 MB 2025-02-14 18:38:32,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23788.03 MB 2025-02-14 18:38:32,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8473.11 MB 2025-02-14 18:38:32,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36828.09 MB 2025-02-14 18:38:32,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 18:38:32,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:38:32,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34772.80 MB 2025-02-14 18:38:32,751 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:38:32,751 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:38:32,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:38:32,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:38:32,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:38:32,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:38:32,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23788.03 MB 2025-02-14 18:38:32,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32227.05 MB 2025-02-14 18:38:32,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:38:32,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36828.09 MB 2025-02-14 18:38:32,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45218.79 MB 2025-02-14 18:38:32,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:38:32,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32227.05 MB 2025-02-14 18:38:32,916 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:38:32,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:38:32,918 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:38:32,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:38:32,919 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:38:32,923 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:38:32,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:38:32,925 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:38:32,925 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:39:13,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:39:13,832 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:39:13,840 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:39:13,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:39:13,845 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1968, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:39:13,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:39:13,847 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1968, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:39:44,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:39:44,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:39:44,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.55 seconds 2025-02-14 18:39:44,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:44,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26682.05 MB 2025-02-14 18:39:44,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33646.69 MB 2025-02-14 18:39:44,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6964.64 MB 2025-02-14 18:39:44,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57803.80 MB 2025-02-14 18:39:44,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37664.85 MB 2025-02-14 18:39:44,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20138.95 MB 2025-02-14 18:39:44,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42496.01 MB 2025-02-14 18:39:44,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:39:44,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:39:44,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 18:39:44,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:44,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33646.69 MB 2025-02-14 18:39:44,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.87 MB 2025-02-14 18:39:44,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7637.82 MB 2025-02-14 18:39:44,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37664.85 MB 2025-02-14 18:39:44,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63346.57 MB 2025-02-14 18:39:44,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25681.72 MB 2025-02-14 18:39:44,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53583.27 MB 2025-02-14 18:39:46,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:39:46,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:39:46,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 18:39:46,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:46,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26008.87 MB 2025-02-14 18:39:46,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26539.71 MB 2025-02-14 18:39:46,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:39:46,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63346.57 MB 2025-02-14 18:39:46,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32115.79 MB 2025-02-14 18:39:46,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31230.79 MB 2025-02-14 18:39:46,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30518.26 MB 2025-02-14 18:39:46,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:39:46,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:39:46,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:39:46,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:46,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26539.71 MB 2025-02-14 18:39:46,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28429.24 MB 2025-02-14 18:39:46,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:39:46,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 18:39:46,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32115.79 MB 2025-02-14 18:39:46,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:39:46,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29846.67 MB 2025-02-14 18:39:46,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:39:46,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:39:46,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:39:46,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:46,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28429.24 MB 2025-02-14 18:39:46,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30671.10 MB 2025-02-14 18:39:46,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:39:46,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 18:39:46,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37778.10 MB 2025-02-14 18:39:46,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:39:46,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36215.38 MB 2025-02-14 18:39:46,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:39:46,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:39:46,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:39:46,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:46,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26539.71 MB 2025-02-14 18:39:46,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30671.10 MB 2025-02-14 18:39:46,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:39:46,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 18:39:46,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37778.10 MB 2025-02-14 18:39:46,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:39:46,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36215.38 MB 2025-02-14 18:39:46,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:39:46,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:39:46,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:39:46,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:46,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32204.64 MB 2025-02-14 18:39:46,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32971.64 MB 2025-02-14 18:39:46,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:39:46,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37778.10 MB 2025-02-14 18:39:46,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 18:39:46,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 18:39:46,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33679.43 MB 2025-02-14 18:39:46,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:39:46,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:39:46,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:39:46,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:46,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33384.53 MB 2025-02-14 18:39:46,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33613.57 MB 2025-02-14 18:39:46,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-14 18:39:46,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38191.24 MB 2025-02-14 18:39:46,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 18:39:46,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:39:46,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33836.82 MB 2025-02-14 18:39:46,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:39:46,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:39:46,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.04 seconds 2025-02-14 18:39:46,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:46,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19825.38 MB 2025-02-14 18:39:46,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33814.20 MB 2025-02-14 18:39:46,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13988.82 MB 2025-02-14 18:39:46,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57803.80 MB 2025-02-14 18:39:46,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 18:39:46,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19612.57 MB 2025-02-14 18:39:46,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33836.82 MB 2025-02-14 18:39:47,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:39:47,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:39:47,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:39:47,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:47,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33814.20 MB 2025-02-14 18:39:47,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24822.91 MB 2025-02-14 18:39:47,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8991.29 MB 2025-02-14 18:39:47,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38191.24 MB 2025-02-14 18:39:47,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 18:39:47,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:39:47,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36320.34 MB 2025-02-14 18:39:47,182 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 18:39:47,183 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:39:47,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:39:47,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:39:47,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:39:47,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:39:47,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24822.91 MB 2025-02-14 18:39:47,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33243.02 MB 2025-02-14 18:39:47,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.11 MB 2025-02-14 18:39:47,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38191.24 MB 2025-02-14 18:39:47,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42377.15 MB 2025-02-14 18:39:47,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-14 18:39:47,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33243.02 MB 2025-02-14 18:39:47,345 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 18:39:47,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:39:47,347 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:39:47,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:39:47,348 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:39:47,353 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:39:47,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:39:47,354 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:39:47,354 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:41:24,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:41:24,765 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:41:24,770 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:41:24,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:41:24,774 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1003, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:41:24,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:41:24,775 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1003, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:41:40,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:41:40,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:41:40,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.42 seconds 2025-02-14 18:41:40,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:40,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19957.77 MB 2025-02-14 18:41:40,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.25 MB 2025-02-14 18:41:40,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3550.48 MB 2025-02-14 18:41:40,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50748.98 MB 2025-02-14 18:41:40,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25855.79 MB 2025-02-14 18:41:40,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24893.19 MB 2025-02-14 18:41:40,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32374.35 MB 2025-02-14 18:41:40,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:41:40,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:41:40,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:41:40,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:40,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.25 MB 2025-02-14 18:41:40,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20993.18 MB 2025-02-14 18:41:40,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2515.07 MB 2025-02-14 18:41:40,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25855.79 MB 2025-02-14 18:41:40,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41634.76 MB 2025-02-14 18:41:40,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15778.97 MB 2025-02-14 18:41:40,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34530.82 MB 2025-02-14 18:41:42,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:41:42,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:41:42,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 18:41:42,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20993.18 MB 2025-02-14 18:41:42,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21524.03 MB 2025-02-14 18:41:42,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:41:42,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41634.76 MB 2025-02-14 18:41:42,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25138.56 MB 2025-02-14 18:41:42,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16496.20 MB 2025-02-14 18:41:42,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25503.61 MB 2025-02-14 18:41:42,217 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:41:42,217 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:41:42,217 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:41:42,217 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,217 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21524.03 MB 2025-02-14 18:41:42,217 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23413.56 MB 2025-02-14 18:41:42,217 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:41:42,217 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25138.56 MB 2025-02-14 18:41:42,217 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27026.00 MB 2025-02-14 18:41:42,217 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 18:41:42,217 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24830.99 MB 2025-02-14 18:41:42,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:41:42,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:41:42,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:41:42,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23413.56 MB 2025-02-14 18:41:42,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25655.42 MB 2025-02-14 18:41:42,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:41:42,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27026.00 MB 2025-02-14 18:41:42,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33160.17 MB 2025-02-14 18:41:42,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:41:42,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.70 MB 2025-02-14 18:41:42,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:41:42,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:41:42,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 18:41:42,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21524.03 MB 2025-02-14 18:41:42,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25655.42 MB 2025-02-14 18:41:42,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:41:42,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25138.56 MB 2025-02-14 18:41:42,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33160.17 MB 2025-02-14 18:41:42,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 18:41:42,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.70 MB 2025-02-14 18:41:42,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:41:42,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:41:42,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:41:42,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27188.96 MB 2025-02-14 18:41:42,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27955.96 MB 2025-02-14 18:41:42,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:41:42,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33160.17 MB 2025-02-14 18:41:42,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-14 18:41:42,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 18:41:42,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28663.75 MB 2025-02-14 18:41:42,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:41:42,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:41:42,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:41:42,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28368.85 MB 2025-02-14 18:41:42,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28596.63 MB 2025-02-14 18:41:42,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.78 MB 2025-02-14 18:41:42,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33571.21 MB 2025-02-14 18:41:42,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-14 18:41:42,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:41:42,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28784.60 MB 2025-02-14 18:41:42,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:41:42,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:41:42,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.84 seconds 2025-02-14 18:41:42,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16463.24 MB 2025-02-14 18:41:42,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28796.92 MB 2025-02-14 18:41:42,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12333.68 MB 2025-02-14 18:41:42,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50748.98 MB 2025-02-14 18:41:42,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-14 18:41:42,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17177.77 MB 2025-02-14 18:41:42,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28796.92 MB 2025-02-14 18:41:42,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:41:42,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:41:42,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:41:42,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28796.92 MB 2025-02-14 18:41:42,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21455.63 MB 2025-02-14 18:41:42,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7341.28 MB 2025-02-14 18:41:42,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33571.21 MB 2025-02-14 18:41:42,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33571.21 MB 2025-02-14 18:41:42,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:41:42,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31298.95 MB 2025-02-14 18:41:42,964 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-14 18:41:42,965 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:41:42,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:41:42,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:41:42,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:41:42,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:41:42,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21455.63 MB 2025-02-14 18:41:42,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29861.30 MB 2025-02-14 18:41:42,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-14 18:41:42,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33571.21 MB 2025-02-14 18:41:42,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41930.46 MB 2025-02-14 18:41:42,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 18:41:42,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29861.30 MB 2025-02-14 18:41:43,141 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-14 18:41:43,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:41:43,142 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:41:43,143 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:41:43,143 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:41:43,148 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:41:43,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:41:43,150 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:41:43,150 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:43:30,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:43:30,188 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:43:30,193 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:43:30,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:43:30,197 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2114, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:43:30,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:43:30,198 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2114, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:44:02,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:44:02,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:44:02,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.46 seconds 2025-02-14 18:44:02,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:02,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27699.40 MB 2025-02-14 18:44:02,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35180.72 MB 2025-02-14 18:44:02,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7481.33 MB 2025-02-14 18:44:02,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50289.70 MB 2025-02-14 18:44:02,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38138.81 MB 2025-02-14 18:44:02,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12150.90 MB 2025-02-14 18:44:02,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44192.84 MB 2025-02-14 18:44:02,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:44:02,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:44:02,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 18:44:02,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:02,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35180.72 MB 2025-02-14 18:44:02,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26768.93 MB 2025-02-14 18:44:02,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8411.80 MB 2025-02-14 18:44:02,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38138.81 MB 2025-02-14 18:44:02,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67954.02 MB 2025-02-14 18:44:02,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29815.21 MB 2025-02-14 18:44:02,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55780.36 MB 2025-02-14 18:44:04,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:44:04,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:44:04,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 18:44:04,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:04,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26768.93 MB 2025-02-14 18:44:04,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27299.77 MB 2025-02-14 18:44:04,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:44:04,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67954.02 MB 2025-02-14 18:44:04,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33489.42 MB 2025-02-14 18:44:04,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34464.60 MB 2025-02-14 18:44:04,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31278.31 MB 2025-02-14 18:44:04,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:44:04,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:44:04,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:44:04,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:04,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27299.77 MB 2025-02-14 18:44:04,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29189.30 MB 2025-02-14 18:44:04,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:44:04,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33489.42 MB 2025-02-14 18:44:04,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33489.42 MB 2025-02-14 18:44:04,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:44:04,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30606.73 MB 2025-02-14 18:44:04,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:44:04,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:44:04,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:44:04,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:04,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29189.30 MB 2025-02-14 18:44:04,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31431.16 MB 2025-02-14 18:44:04,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:44:04,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33489.42 MB 2025-02-14 18:44:04,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38679.87 MB 2025-02-14 18:44:04,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:44:04,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36975.44 MB 2025-02-14 18:44:04,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:44:04,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:44:04,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:44:04,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:04,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27299.77 MB 2025-02-14 18:44:04,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31431.16 MB 2025-02-14 18:44:04,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:44:04,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33489.42 MB 2025-02-14 18:44:04,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38679.87 MB 2025-02-14 18:44:04,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:44:04,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36975.44 MB 2025-02-14 18:44:05,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:44:05,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:44:05,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:44:05,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:05,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32964.70 MB 2025-02-14 18:44:05,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33731.70 MB 2025-02-14 18:44:05,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:44:05,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38679.87 MB 2025-02-14 18:44:05,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39093.01 MB 2025-02-14 18:44:05,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 18:44:05,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34439.49 MB 2025-02-14 18:44:05,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:44:05,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:44:05,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:44:05,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:05,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34144.59 MB 2025-02-14 18:44:05,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34373.05 MB 2025-02-14 18:44:05,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-14 18:44:05,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39093.01 MB 2025-02-14 18:44:05,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39093.01 MB 2025-02-14 18:44:05,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:44:05,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34594.38 MB 2025-02-14 18:44:05,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:44:05,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:44:05,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.95 seconds 2025-02-14 18:44:05,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:05,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20334.05 MB 2025-02-14 18:44:05,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34573.58 MB 2025-02-14 18:44:05,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14239.53 MB 2025-02-14 18:44:05,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50289.70 MB 2025-02-14 18:44:05,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39093.01 MB 2025-02-14 18:44:05,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11196.69 MB 2025-02-14 18:44:05,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34594.38 MB 2025-02-14 18:44:05,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:44:05,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:44:05,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:44:05,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:05,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34573.58 MB 2025-02-14 18:44:05,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25330.06 MB 2025-02-14 18:44:05,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9243.52 MB 2025-02-14 18:44:05,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39093.01 MB 2025-02-14 18:44:05,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39093.01 MB 2025-02-14 18:44:05,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:44:05,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37078.49 MB 2025-02-14 18:44:05,440 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 18:44:05,441 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:44:05,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:44:05,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:44:05,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:44:05,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:44:05,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25330.06 MB 2025-02-14 18:44:05,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33746.66 MB 2025-02-14 18:44:05,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 18:44:05,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39093.01 MB 2025-02-14 18:44:05,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47460.65 MB 2025-02-14 18:44:05,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 18:44:05,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33746.66 MB 2025-02-14 18:44:05,609 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 18:44:05,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:44:05,611 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:44:05,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:44:05,612 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:44:05,616 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:44:05,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:44:05,617 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:44:05,617 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:44:58,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:44:58,570 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:44:58,574 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:44:58,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:44:58,578 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2518, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:44:58,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:44:58,579 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2518, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:45:37,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:45:37,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:45:37,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.96 seconds 2025-02-14 18:45:37,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:37,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30517.67 MB 2025-02-14 18:45:37,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39428.73 MB 2025-02-14 18:45:37,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8911.06 MB 2025-02-14 18:45:37,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73377.25 MB 2025-02-14 18:45:37,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44574.97 MB 2025-02-14 18:45:37,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28802.29 MB 2025-02-14 18:45:37,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48369.26 MB 2025-02-14 18:45:37,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:45:37,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:45:37,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:45:37,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:37,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39428.73 MB 2025-02-14 18:45:37,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28870.37 MB 2025-02-14 18:45:37,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10558.36 MB 2025-02-14 18:45:37,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44574.97 MB 2025-02-14 18:45:37,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75694.60 MB 2025-02-14 18:45:37,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 31119.64 MB 2025-02-14 18:45:37,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65307.44 MB 2025-02-14 18:45:39,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:45:39,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:45:39,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 18:45:39,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:39,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28870.37 MB 2025-02-14 18:45:39,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29401.21 MB 2025-02-14 18:45:39,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:45:39,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75694.60 MB 2025-02-14 18:45:39,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32621.20 MB 2025-02-14 18:45:39,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -43073.40 MB 2025-02-14 18:45:39,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33380.80 MB 2025-02-14 18:45:39,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:45:39,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:45:39,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:45:39,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:39,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29401.21 MB 2025-02-14 18:45:39,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31290.75 MB 2025-02-14 18:45:39,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:45:39,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32621.20 MB 2025-02-14 18:45:39,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34508.64 MB 2025-02-14 18:45:39,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 18:45:39,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32708.17 MB 2025-02-14 18:45:39,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:45:39,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:45:39,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:45:39,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:39,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31290.75 MB 2025-02-14 18:45:39,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33532.60 MB 2025-02-14 18:45:39,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:45:39,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34508.64 MB 2025-02-14 18:45:39,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40642.81 MB 2025-02-14 18:45:39,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:45:39,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39076.88 MB 2025-02-14 18:45:39,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:45:39,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:45:39,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:45:39,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:39,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29401.21 MB 2025-02-14 18:45:39,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33532.60 MB 2025-02-14 18:45:39,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:45:39,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32621.20 MB 2025-02-14 18:45:39,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40642.81 MB 2025-02-14 18:45:39,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 18:45:39,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39076.88 MB 2025-02-14 18:45:40,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:45:40,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:45:40,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:45:40,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:40,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35066.14 MB 2025-02-14 18:45:40,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35833.15 MB 2025-02-14 18:45:40,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:45:40,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40642.81 MB 2025-02-14 18:45:40,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41060.14 MB 2025-02-14 18:45:40,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:45:40,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36540.93 MB 2025-02-14 18:45:40,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:45:40,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:45:40,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:45:40,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:40,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36246.03 MB 2025-02-14 18:45:40,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36474.63 MB 2025-02-14 18:45:40,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.59 MB 2025-02-14 18:45:40,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41060.14 MB 2025-02-14 18:45:40,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41060.14 MB 2025-02-14 18:45:40,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:45:40,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36691.55 MB 2025-02-14 18:45:40,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:45:40,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:45:40,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.50 seconds 2025-02-14 18:45:40,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:40,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21743.19 MB 2025-02-14 18:45:40,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36675.13 MB 2025-02-14 18:45:40,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14931.94 MB 2025-02-14 18:45:40,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64602.77 MB 2025-02-14 18:45:40,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41060.14 MB 2025-02-14 18:45:40,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23542.63 MB 2025-02-14 18:45:40,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36691.55 MB 2025-02-14 18:45:40,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:45:40,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:45:40,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:45:40,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:40,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36675.13 MB 2025-02-14 18:45:40,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26738.82 MB 2025-02-14 18:45:40,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9936.32 MB 2025-02-14 18:45:40,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41060.14 MB 2025-02-14 18:45:40,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41060.14 MB 2025-02-14 18:45:40,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:45:40,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39179.74 MB 2025-02-14 18:45:40,370 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 18:45:40,371 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:45:40,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:45:40,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:45:40,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:45:40,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:45:40,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26738.82 MB 2025-02-14 18:45:40,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35153.77 MB 2025-02-14 18:45:40,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 18:45:40,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41060.14 MB 2025-02-14 18:45:40,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45243.96 MB 2025-02-14 18:45:40,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 18:45:40,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35153.77 MB 2025-02-14 18:45:40,534 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 18:45:40,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:45:40,535 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:45:40,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:45:40,536 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:45:40,541 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:45:40,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:45:40,542 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:45:40,542 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 18:45:49,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:45:49,745 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:45:49,750 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:45:49,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:45:49,753 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1421, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:45:49,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:45:49,754 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1421, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:46:11,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:46:11,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:46:11,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.19 seconds 2025-02-14 18:46:11,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:11,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22870.46 MB 2025-02-14 18:46:11,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27899.43 MB 2025-02-14 18:46:11,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5028.97 MB 2025-02-14 18:46:11,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53611.59 MB 2025-02-14 18:46:11,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35695.62 MB 2025-02-14 18:46:11,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17915.97 MB 2025-02-14 18:46:11,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36871.68 MB 2025-02-14 18:46:12,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:46:12,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:46:12,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:46:12,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:12,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27899.43 MB 2025-02-14 18:46:12,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23165.19 MB 2025-02-14 18:46:12,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4734.25 MB 2025-02-14 18:46:12,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35695.62 MB 2025-02-14 18:46:12,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47399.83 MB 2025-02-14 18:46:12,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11704.21 MB 2025-02-14 18:46:12,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42395.30 MB 2025-02-14 18:46:13,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:46:13,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:46:13,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 18:46:13,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:13,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23165.19 MB 2025-02-14 18:46:13,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23696.03 MB 2025-02-14 18:46:13,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:46:13,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47399.83 MB 2025-02-14 18:46:13,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30666.65 MB 2025-02-14 18:46:13,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16733.18 MB 2025-02-14 18:46:13,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27674.57 MB 2025-02-14 18:46:13,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:46:13,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:46:13,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:46:13,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:13,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23696.03 MB 2025-02-14 18:46:13,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25585.56 MB 2025-02-14 18:46:13,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:46:13,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30666.65 MB 2025-02-14 18:46:13,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30666.65 MB 2025-02-14 18:46:13,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:46:13,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27002.99 MB 2025-02-14 18:46:14,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:46:14,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:46:14,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:46:14,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:14,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25585.56 MB 2025-02-14 18:46:14,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27827.42 MB 2025-02-14 18:46:14,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:46:14,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30666.65 MB 2025-02-14 18:46:14,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35857.10 MB 2025-02-14 18:46:14,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:46:14,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33371.70 MB 2025-02-14 18:46:14,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:46:14,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:46:14,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:46:14,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:14,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23696.03 MB 2025-02-14 18:46:14,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27827.42 MB 2025-02-14 18:46:14,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:46:14,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30666.65 MB 2025-02-14 18:46:14,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35857.10 MB 2025-02-14 18:46:14,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:46:14,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33371.70 MB 2025-02-14 18:46:14,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:46:14,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:46:14,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 18:46:14,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:14,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29360.96 MB 2025-02-14 18:46:14,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30127.96 MB 2025-02-14 18:46:14,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:46:14,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35857.10 MB 2025-02-14 18:46:14,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36272.34 MB 2025-02-14 18:46:14,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:46:14,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30835.75 MB 2025-02-14 18:46:14,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:46:14,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:46:14,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:46:14,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:14,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30540.85 MB 2025-02-14 18:46:14,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30769.20 MB 2025-02-14 18:46:14,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 18:46:14,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36272.34 MB 2025-02-14 18:46:14,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36272.34 MB 2025-02-14 18:46:14,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:46:14,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30980.88 MB 2025-02-14 18:46:14,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:46:14,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:46:14,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.64 seconds 2025-02-14 18:46:14,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:14,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17919.58 MB 2025-02-14 18:46:14,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30970.10 MB 2025-02-14 18:46:14,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13050.51 MB 2025-02-14 18:46:14,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53611.59 MB 2025-02-14 18:46:14,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36272.34 MB 2025-02-14 18:46:14,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17339.25 MB 2025-02-14 18:46:14,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30980.88 MB 2025-02-14 18:46:14,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:46:14,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:46:14,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:46:14,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:14,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30970.10 MB 2025-02-14 18:46:14,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22921.31 MB 2025-02-14 18:46:14,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8048.79 MB 2025-02-14 18:46:14,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36272.34 MB 2025-02-14 18:46:14,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36272.34 MB 2025-02-14 18:46:14,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:46:14,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33479.61 MB 2025-02-14 18:46:14,687 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 18:46:14,687 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:46:14,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:46:14,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:46:14,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:46:14,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:46:14,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22921.31 MB 2025-02-14 18:46:14,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31352.77 MB 2025-02-14 18:46:14,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 18:46:14,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36272.34 MB 2025-02-14 18:46:14,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44656.75 MB 2025-02-14 18:46:14,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 18:46:14,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31352.77 MB 2025-02-14 18:46:14,850 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 18:46:14,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:46:14,852 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:46:14,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:46:14,853 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:46:14,857 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:46:14,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:46:14,858 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:46:14,858 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:47:03,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:03,071 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:47:03,076 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:47:03,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:03,079 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 178, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:47:03,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:03,080 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 178, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:47:05,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:47:05,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:47:05,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.77 seconds 2025-02-14 18:47:05,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:05,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14209.04 MB 2025-02-14 18:47:05,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14838.97 MB 2025-02-14 18:47:05,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 629.93 MB 2025-02-14 18:47:05,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53041.17 MB 2025-02-14 18:47:05,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:47:05,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33508.29 MB 2025-02-14 18:47:05,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23680.41 MB 2025-02-14 18:47:05,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:47:05,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:47:05,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:47:05,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:05,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14838.97 MB 2025-02-14 18:47:05,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15137.15 MB 2025-02-14 18:47:05,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.18 MB 2025-02-14 18:47:05,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:47:05,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:47:05,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:47:05,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17332.26 MB 2025-02-14 18:47:06,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:47:06,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:47:06,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 18:47:06,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:06,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15137.15 MB 2025-02-14 18:47:06,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15372.05 MB 2025-02-14 18:47:06,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 18:47:06,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:47:06,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:47:06,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:47:06,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19307.84 MB 2025-02-14 18:47:06,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:47:06,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:47:06,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:47:06,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:06,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15371.98 MB 2025-02-14 18:47:06,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16207.90 MB 2025-02-14 18:47:06,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 18:47:06,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:47:06,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:47:06,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:47:06,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16835.11 MB 2025-02-14 18:47:06,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:47:06,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:47:06,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:47:06,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:06,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16207.90 MB 2025-02-14 18:47:06,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17199.95 MB 2025-02-14 18:47:06,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 18:47:06,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:47:06,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21000.88 MB 2025-02-14 18:47:06,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1468.01 MB 2025-02-14 18:47:06,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19654.48 MB 2025-02-14 18:47:06,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:47:06,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:47:06,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:47:06,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:06,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15371.98 MB 2025-02-14 18:47:06,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17199.95 MB 2025-02-14 18:47:06,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 18:47:06,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:47:06,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21000.88 MB 2025-02-14 18:47:06,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1468.01 MB 2025-02-14 18:47:06,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19654.48 MB 2025-02-14 18:47:06,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:47:06,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:47:06,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:47:06,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:06,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17878.55 MB 2025-02-14 18:47:06,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18219.16 MB 2025-02-14 18:47:06,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.61 MB 2025-02-14 18:47:06,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21000.88 MB 2025-02-14 18:47:06,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21181.24 MB 2025-02-14 18:47:06,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 18:47:06,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18537.75 MB 2025-02-14 18:47:06,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:47:06,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:47:06,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:47:06,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:06,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18401.87 MB 2025-02-14 18:47:06,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18629.62 MB 2025-02-14 18:47:06,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.75 MB 2025-02-14 18:47:06,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21181.24 MB 2025-02-14 18:47:06,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21181.24 MB 2025-02-14 18:47:06,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:47:06,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18644.26 MB 2025-02-14 18:47:06,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:47:06,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:47:06,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.82 seconds 2025-02-14 18:47:06,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:06,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13588.87 MB 2025-02-14 18:47:06,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18830.17 MB 2025-02-14 18:47:06,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5241.30 MB 2025-02-14 18:47:06,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53041.17 MB 2025-02-14 18:47:06,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21181.24 MB 2025-02-14 18:47:06,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31859.93 MB 2025-02-14 18:47:06,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18830.17 MB 2025-02-14 18:47:07,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:47:07,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:47:07,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:47:07,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:07,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18830.17 MB 2025-02-14 18:47:07,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17534.07 MB 2025-02-14 18:47:07,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1296.10 MB 2025-02-14 18:47:07,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21181.24 MB 2025-02-14 18:47:07,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21181.24 MB 2025-02-14 18:47:07,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:47:07,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19064.94 MB 2025-02-14 18:47:07,189 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 18:47:07,190 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:47:07,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:47:07,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:47:07,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:47:07,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:47:07,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17534.07 MB 2025-02-14 18:47:07,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25951.81 MB 2025-02-14 18:47:07,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-14 18:47:07,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21181.24 MB 2025-02-14 18:47:07,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29548.87 MB 2025-02-14 18:47:07,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 18:47:07,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25951.81 MB 2025-02-14 18:47:07,351 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 18:47:07,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:07,353 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:47:07,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:07,354 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:47:07,358 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:47:07,359 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:07,359 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:47:07,359 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:47:49,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:49,416 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:47:49,421 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:47:49,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:49,425 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1048, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:47:49,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:47:49,426 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1048, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:48:05,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:48:05,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:48:05,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.08 seconds 2025-02-14 18:48:05,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:05,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20271.34 MB 2025-02-14 18:48:05,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23981.20 MB 2025-02-14 18:48:05,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3709.86 MB 2025-02-14 18:48:05,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-14 18:48:05,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30190.60 MB 2025-02-14 18:48:05,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7725.91 MB 2025-02-14 18:48:05,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32913.60 MB 2025-02-14 18:48:05,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:48:05,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:48:05,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:48:05,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:05,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23981.20 MB 2025-02-14 18:48:05,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21226.08 MB 2025-02-14 18:48:05,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2755.12 MB 2025-02-14 18:48:05,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30190.60 MB 2025-02-14 18:48:05,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37297.85 MB 2025-02-14 18:48:05,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7107.25 MB 2025-02-14 18:48:05,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33923.21 MB 2025-02-14 18:48:07,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:48:07,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:48:07,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 18:48:07,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:07,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21226.08 MB 2025-02-14 18:48:07,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21756.92 MB 2025-02-14 18:48:07,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:48:07,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37297.85 MB 2025-02-14 18:48:07,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27896.32 MB 2025-02-14 18:48:07,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9401.53 MB 2025-02-14 18:48:07,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25735.46 MB 2025-02-14 18:48:07,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:48:07,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:48:07,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:48:07,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:07,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21756.92 MB 2025-02-14 18:48:07,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23646.45 MB 2025-02-14 18:48:07,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:48:07,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27896.32 MB 2025-02-14 18:48:07,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27896.32 MB 2025-02-14 18:48:07,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:48:07,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25063.88 MB 2025-02-14 18:48:07,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:48:07,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:48:07,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:48:07,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:07,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23646.45 MB 2025-02-14 18:48:07,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25888.31 MB 2025-02-14 18:48:07,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:48:07,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27896.32 MB 2025-02-14 18:48:07,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33086.77 MB 2025-02-14 18:48:07,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:48:07,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31432.59 MB 2025-02-14 18:48:07,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:48:07,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:48:07,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:48:07,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:07,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21756.92 MB 2025-02-14 18:48:07,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25888.31 MB 2025-02-14 18:48:07,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:48:07,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27896.32 MB 2025-02-14 18:48:07,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33086.77 MB 2025-02-14 18:48:07,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:48:07,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31432.59 MB 2025-02-14 18:48:07,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:48:07,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:48:07,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:48:07,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:07,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27421.85 MB 2025-02-14 18:48:07,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28188.85 MB 2025-02-14 18:48:07,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:48:07,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33086.77 MB 2025-02-14 18:48:07,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33504.10 MB 2025-02-14 18:48:07,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:48:07,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28896.64 MB 2025-02-14 18:48:07,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:48:07,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:48:07,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:48:07,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:07,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28601.74 MB 2025-02-14 18:48:07,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28832.14 MB 2025-02-14 18:48:07,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.40 MB 2025-02-14 18:48:07,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33504.10 MB 2025-02-14 18:48:07,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33504.10 MB 2025-02-14 18:48:07,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:48:07,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29018.58 MB 2025-02-14 18:48:07,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:48:07,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:48:07,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.45 seconds 2025-02-14 18:48:07,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:07,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16620.02 MB 2025-02-14 18:48:07,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29033.21 MB 2025-02-14 18:48:07,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12413.19 MB 2025-02-14 18:48:07,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-14 18:48:07,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33504.10 MB 2025-02-14 18:48:07,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4412.41 MB 2025-02-14 18:48:07,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29033.21 MB 2025-02-14 18:48:08,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:48:08,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:48:08,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:48:08,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:08,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29033.21 MB 2025-02-14 18:48:08,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21624.41 MB 2025-02-14 18:48:08,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7408.80 MB 2025-02-14 18:48:08,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33504.10 MB 2025-02-14 18:48:08,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33504.10 MB 2025-02-14 18:48:08,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:48:08,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31544.88 MB 2025-02-14 18:48:08,159 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:48:08,159 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:48:08,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:48:08,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:48:08,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:48:08,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:48:08,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21624.41 MB 2025-02-14 18:48:08,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30063.43 MB 2025-02-14 18:48:08,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:48:08,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33504.10 MB 2025-02-14 18:48:08,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41894.81 MB 2025-02-14 18:48:08,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:48:08,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30063.43 MB 2025-02-14 18:48:08,322 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:48:08,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:48:08,324 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:48:08,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:48:08,325 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:48:08,329 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:48:08,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:48:08,330 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:48:08,330 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:49:31,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:49:31,949 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:49:31,954 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:49:31,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:49:31,958 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1074, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:49:31,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:49:31,959 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1074, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:49:48,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:49:48,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:49:48,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.46 seconds 2025-02-14 18:49:48,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:48,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20452.51 MB 2025-02-14 18:49:48,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24253.34 MB 2025-02-14 18:49:48,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3800.83 MB 2025-02-14 18:49:48,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54479.81 MB 2025-02-14 18:49:48,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30284.97 MB 2025-02-14 18:49:48,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24194.84 MB 2025-02-14 18:49:48,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33094.78 MB 2025-02-14 18:49:48,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:49:48,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:49:48,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:49:48,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:48,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24253.34 MB 2025-02-14 18:49:48,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21361.24 MB 2025-02-14 18:49:48,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2892.09 MB 2025-02-14 18:49:48,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30284.97 MB 2025-02-14 18:49:48,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37153.14 MB 2025-02-14 18:49:48,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6868.17 MB 2025-02-14 18:49:48,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33936.64 MB 2025-02-14 18:49:50,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:49:50,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:49:50,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 18:49:50,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:50,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21361.24 MB 2025-02-14 18:49:50,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21892.08 MB 2025-02-14 18:49:50,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:49:50,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37153.14 MB 2025-02-14 18:49:50,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27898.41 MB 2025-02-14 18:49:50,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9254.73 MB 2025-02-14 18:49:50,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25870.63 MB 2025-02-14 18:49:50,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:49:50,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:49:50,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:49:50,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:50,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21892.08 MB 2025-02-14 18:49:50,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23781.62 MB 2025-02-14 18:49:50,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:49:50,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27898.41 MB 2025-02-14 18:49:50,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27898.41 MB 2025-02-14 18:49:50,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:49:50,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25199.05 MB 2025-02-14 18:49:50,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:49:50,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:49:50,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:49:50,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:50,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23781.62 MB 2025-02-14 18:49:50,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26023.47 MB 2025-02-14 18:49:50,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:49:50,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27898.41 MB 2025-02-14 18:49:50,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33560.72 MB 2025-02-14 18:49:50,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:49:50,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.76 MB 2025-02-14 18:49:50,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:49:50,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:49:50,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:49:50,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:50,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21892.08 MB 2025-02-14 18:49:50,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26023.47 MB 2025-02-14 18:49:50,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:49:50,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27898.41 MB 2025-02-14 18:49:50,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33560.72 MB 2025-02-14 18:49:50,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 18:49:50,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.76 MB 2025-02-14 18:49:50,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:49:50,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:49:50,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:49:50,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:50,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27557.02 MB 2025-02-14 18:49:50,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28324.02 MB 2025-02-14 18:49:50,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:49:50,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33560.72 MB 2025-02-14 18:49:50,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-14 18:49:50,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:49:50,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29031.81 MB 2025-02-14 18:49:50,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:49:50,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:49:50,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:49:50,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:50,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28736.91 MB 2025-02-14 18:49:50,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28966.51 MB 2025-02-14 18:49:50,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.60 MB 2025-02-14 18:49:50,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-14 18:49:50,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-14 18:49:50,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:49:50,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29188.18 MB 2025-02-14 18:49:50,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:49:50,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:49:50,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.83 seconds 2025-02-14 18:49:50,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:50,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16710.61 MB 2025-02-14 18:49:50,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29167.58 MB 2025-02-14 18:49:50,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12456.97 MB 2025-02-14 18:49:50,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54479.81 MB 2025-02-14 18:49:50,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-14 18:49:50,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20503.86 MB 2025-02-14 18:49:50,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29188.18 MB 2025-02-14 18:49:51,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:49:51,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:49:51,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:49:51,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:51,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29167.58 MB 2025-02-14 18:49:51,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21715.00 MB 2025-02-14 18:49:51,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7452.58 MB 2025-02-14 18:49:51,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-14 18:49:51,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-14 18:49:51,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:49:51,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31679.25 MB 2025-02-14 18:49:51,074 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:49:51,074 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 18:49:51,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:49:51,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:49:51,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:49:51,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:49:51,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21715.00 MB 2025-02-14 18:49:51,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30154.02 MB 2025-02-14 18:49:51,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:49:51,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-14 18:49:51,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42366.66 MB 2025-02-14 18:49:51,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 18:49:51,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30154.02 MB 2025-02-14 18:49:51,238 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:49:51,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:49:51,240 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:49:51,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:49:51,240 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:49:51,245 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:49:51,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:49:51,246 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:49:51,246 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 18:50:01,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:50:01,167 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:50:01,175 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:50:01,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:50:01,182 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1869, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:50:01,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:50:01,184 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1869, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:50:30,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:50:30,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:50:30,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.15 seconds 2025-02-14 18:50:30,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:30,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25992.20 MB 2025-02-14 18:50:30,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32606.61 MB 2025-02-14 18:50:30,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6614.42 MB 2025-02-14 18:50:30,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54951.67 MB 2025-02-14 18:50:30,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37291.56 MB 2025-02-14 18:50:30,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17660.12 MB 2025-02-14 18:50:30,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41579.67 MB 2025-02-14 18:50:30,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:50:30,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:50:30,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 18:50:30,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:30,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32606.61 MB 2025-02-14 18:50:30,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25494.20 MB 2025-02-14 18:50:30,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7112.42 MB 2025-02-14 18:50:30,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37291.56 MB 2025-02-14 18:50:30,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60655.93 MB 2025-02-14 18:50:30,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23364.37 MB 2025-02-14 18:50:30,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51307.11 MB 2025-02-14 18:50:32,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:50:32,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:50:32,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 18:50:32,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:32,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25494.20 MB 2025-02-14 18:50:32,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26025.04 MB 2025-02-14 18:50:32,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:50:32,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60655.93 MB 2025-02-14 18:50:32,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32092.72 MB 2025-02-14 18:50:32,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28563.21 MB 2025-02-14 18:50:32,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30003.59 MB 2025-02-14 18:50:32,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:50:32,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:50:32,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:50:32,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:32,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26025.04 MB 2025-02-14 18:50:32,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27914.57 MB 2025-02-14 18:50:32,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:50:32,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 18:50:32,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32092.72 MB 2025-02-14 18:50:32,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:50:32,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29332.00 MB 2025-02-14 18:50:32,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:50:32,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:50:32,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:50:32,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:32,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27914.57 MB 2025-02-14 18:50:32,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30156.43 MB 2025-02-14 18:50:32,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:50:32,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 18:50:32,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37283.17 MB 2025-02-14 18:50:32,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:50:32,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35700.71 MB 2025-02-14 18:50:32,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:50:32,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:50:32,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 18:50:32,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:32,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26025.04 MB 2025-02-14 18:50:32,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30156.43 MB 2025-02-14 18:50:32,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:50:32,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 18:50:32,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37283.17 MB 2025-02-14 18:50:32,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:50:32,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35700.71 MB 2025-02-14 18:50:32,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:50:32,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:50:32,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:50:32,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:32,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31689.97 MB 2025-02-14 18:50:32,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32456.97 MB 2025-02-14 18:50:32,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:50:32,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37283.17 MB 2025-02-14 18:50:32,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-14 18:50:32,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:50:32,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33164.76 MB 2025-02-14 18:50:32,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:50:32,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:50:32,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:50:32,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:32,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32869.86 MB 2025-02-14 18:50:32,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33097.91 MB 2025-02-14 18:50:32,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-14 18:50:32,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-14 18:50:32,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-14 18:50:32,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:50:32,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33302.25 MB 2025-02-14 18:50:32,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:50:32,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:50:32,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.66 seconds 2025-02-14 18:50:32,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:32,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19480.45 MB 2025-02-14 18:50:32,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33298.40 MB 2025-02-14 18:50:32,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13817.95 MB 2025-02-14 18:50:32,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54951.67 MB 2025-02-14 18:50:32,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-14 18:50:32,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17253.27 MB 2025-02-14 18:50:32,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33302.25 MB 2025-02-14 18:50:33,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:50:33,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:50:33,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:50:33,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:33,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33298.40 MB 2025-02-14 18:50:33,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24468.22 MB 2025-02-14 18:50:33,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8830.18 MB 2025-02-14 18:50:33,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-14 18:50:33,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37698.40 MB 2025-02-14 18:50:33,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:50:33,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35796.24 MB 2025-02-14 18:50:33,131 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 18:50:33,131 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:50:33,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:50:33,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:50:33,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:50:33,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:50:33,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24468.22 MB 2025-02-14 18:50:33,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32860.81 MB 2025-02-14 18:50:33,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-14 18:50:33,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37698.40 MB 2025-02-14 18:50:33,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41869.64 MB 2025-02-14 18:50:33,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 18:50:33,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32860.81 MB 2025-02-14 18:50:33,296 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 18:50:33,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:50:33,297 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:50:33,298 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:50:33,298 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:50:33,303 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:50:33,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:50:33,304 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:50:33,304 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:51:50,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:51:50,492 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:51:50,500 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:51:50,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:51:50,508 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 151, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:51:50,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:51:50,509 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 151, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:51:52,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:51:52,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:51:52,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.37 seconds 2025-02-14 18:51:52,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:52,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14020.90 MB 2025-02-14 18:51:52,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14555.28 MB 2025-02-14 18:51:52,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 534.38 MB 2025-02-14 18:51:52,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54385.44 MB 2025-02-14 18:51:52,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 18:51:52,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34854.67 MB 2025-02-14 18:51:52,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23492.27 MB 2025-02-14 18:51:52,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:51:52,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:51:52,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:51:52,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:52,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14555.28 MB 2025-02-14 18:51:52,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14772.05 MB 2025-02-14 18:51:52,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.77 MB 2025-02-14 18:51:52,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 18:51:52,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 18:51:52,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:51:52,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16616.79 MB 2025-02-14 18:51:53,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:51:53,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:51:53,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 18:51:53,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:53,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14772.05 MB 2025-02-14 18:51:53,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14964.48 MB 2025-02-14 18:51:53,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 18:51:53,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 18:51:53,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 18:51:53,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:51:53,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18941.70 MB 2025-02-14 18:51:53,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:51:53,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:51:53,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:51:53,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:53,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14964.41 MB 2025-02-14 18:51:53,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15649.20 MB 2025-02-14 18:51:53,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 18:51:53,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 18:51:53,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 18:51:53,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:51:53,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16163.03 MB 2025-02-14 18:51:53,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:51:53,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:51:53,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:51:53,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:53,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15649.20 MB 2025-02-14 18:51:53,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16461.92 MB 2025-02-14 18:51:53,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 18:51:53,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 18:51:53,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 18:51:53,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:51:53,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18471.68 MB 2025-02-14 18:51:53,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:51:53,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:51:53,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:51:53,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:53,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14964.41 MB 2025-02-14 18:51:53,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16461.92 MB 2025-02-14 18:51:53,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 18:51:53,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 18:51:53,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 18:51:53,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:51:53,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18471.68 MB 2025-02-14 18:51:53,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:51:53,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:51:53,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:51:53,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:53,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17017.83 MB 2025-02-14 18:51:53,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17295.86 MB 2025-02-14 18:51:53,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 18:51:53,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 18:51:53,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 18:51:53,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 18:51:53,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17561.77 MB 2025-02-14 18:51:53,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:51:53,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:51:53,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:51:53,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:53,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17445.54 MB 2025-02-14 18:51:53,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17671.29 MB 2025-02-14 18:51:53,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.75 MB 2025-02-14 18:51:53,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19679.67 MB 2025-02-14 18:51:53,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 18:51:53,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:51:53,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17675.72 MB 2025-02-14 18:51:53,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:51:53,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:51:53,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.24 seconds 2025-02-14 18:51:53,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:53,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13494.80 MB 2025-02-14 18:51:53,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17872.29 MB 2025-02-14 18:51:53,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4377.49 MB 2025-02-14 18:51:53,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54385.44 MB 2025-02-14 18:51:53,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 18:51:53,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34705.77 MB 2025-02-14 18:51:53,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17872.29 MB 2025-02-14 18:51:54,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:51:54,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:51:54,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:51:54,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:54,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17872.29 MB 2025-02-14 18:51:54,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17294.01 MB 2025-02-14 18:51:54,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -578.29 MB 2025-02-14 18:51:54,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19679.67 MB 2025-02-14 18:51:54,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 18:51:54,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:51:54,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18977.02 MB 2025-02-14 18:51:54,041 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 18:51:54,041 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:51:54,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:51:54,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:51:54,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:51:54,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:51:54,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17294.01 MB 2025-02-14 18:51:54,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25729.60 MB 2025-02-14 18:51:54,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 18:51:54,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19679.67 MB 2025-02-14 18:51:54,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30165.43 MB 2025-02-14 18:51:54,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 18:51:54,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25729.60 MB 2025-02-14 18:51:54,205 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 18:51:54,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:51:54,207 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:51:54,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:51:54,208 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:51:54,212 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:51:54,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:51:54,214 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:51:54,214 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:53:49,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:53:49,092 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:53:49,097 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:53:49,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:53:49,101 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1720, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:53:49,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:53:49,102 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1720, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:54:15,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:54:15,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:54:15,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.24 seconds 2025-02-14 18:54:15,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:15,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24953.94 MB 2025-02-14 18:54:15,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31041.97 MB 2025-02-14 18:54:15,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6088.03 MB 2025-02-14 18:54:15,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38554.04 MB 2025-02-14 18:54:15,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36788.24 MB 2025-02-14 18:54:15,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1765.80 MB 2025-02-14 18:54:15,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39861.94 MB 2025-02-14 18:54:15,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:54:15,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:54:15,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 18:54:15,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:15,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31041.97 MB 2025-02-14 18:54:15,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24719.59 MB 2025-02-14 18:54:15,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6322.38 MB 2025-02-14 18:54:15,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36788.24 MB 2025-02-14 18:54:15,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57789.12 MB 2025-02-14 18:54:15,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21000.88 MB 2025-02-14 18:54:15,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48798.16 MB 2025-02-14 18:54:17,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:54:17,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:54:17,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 18:54:17,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:17,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24719.59 MB 2025-02-14 18:54:17,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25250.44 MB 2025-02-14 18:54:17,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:54:17,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57789.12 MB 2025-02-14 18:54:17,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32115.79 MB 2025-02-14 18:54:17,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25673.33 MB 2025-02-14 18:54:17,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29228.98 MB 2025-02-14 18:54:17,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:54:17,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:54:17,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:54:17,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:17,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.44 MB 2025-02-14 18:54:17,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27139.97 MB 2025-02-14 18:54:17,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:54:17,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 18:54:17,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32115.79 MB 2025-02-14 18:54:17,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:54:17,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28557.40 MB 2025-02-14 18:54:17,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:54:17,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:54:17,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:54:17,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:17,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27139.97 MB 2025-02-14 18:54:17,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29381.83 MB 2025-02-14 18:54:17,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:54:17,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 18:54:17,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36834.38 MB 2025-02-14 18:54:17,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 18:54:17,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34926.11 MB 2025-02-14 18:54:17,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:54:17,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:54:17,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:54:17,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:17,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.44 MB 2025-02-14 18:54:17,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29381.83 MB 2025-02-14 18:54:17,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:54:17,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 18:54:17,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36834.38 MB 2025-02-14 18:54:17,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 18:54:17,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34926.11 MB 2025-02-14 18:54:17,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:54:17,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:54:17,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:54:17,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:17,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30915.37 MB 2025-02-14 18:54:17,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31682.37 MB 2025-02-14 18:54:17,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:54:17,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36834.38 MB 2025-02-14 18:54:17,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37247.52 MB 2025-02-14 18:54:17,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 18:54:17,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32390.16 MB 2025-02-14 18:54:17,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:54:17,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:54:17,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:54:17,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:17,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32095.26 MB 2025-02-14 18:54:17,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32323.56 MB 2025-02-14 18:54:17,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-14 18:54:17,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37247.52 MB 2025-02-14 18:54:17,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37247.52 MB 2025-02-14 18:54:17,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:54:17,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32535.60 MB 2025-02-14 18:54:17,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:54:17,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:54:17,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.66 seconds 2025-02-14 18:54:17,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:17,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18961.32 MB 2025-02-14 18:54:17,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32523.77 MB 2025-02-14 18:54:17,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13562.44 MB 2025-02-14 18:54:17,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38554.04 MB 2025-02-14 18:54:17,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37247.52 MB 2025-02-14 18:54:17,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1306.53 MB 2025-02-14 18:54:17,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32535.60 MB 2025-02-14 18:54:18,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:54:18,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:54:18,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:54:18,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:18,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32523.77 MB 2025-02-14 18:54:18,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23952.65 MB 2025-02-14 18:54:18,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8571.12 MB 2025-02-14 18:54:18,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37247.52 MB 2025-02-14 18:54:18,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37247.52 MB 2025-02-14 18:54:18,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:54:18,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35024.68 MB 2025-02-14 18:54:18,049 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 18:54:18,049 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:54:18,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:54:18,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:54:18,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:54:18,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:54:18,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23952.65 MB 2025-02-14 18:54:18,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32356.21 MB 2025-02-14 18:54:18,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 18:54:18,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37247.52 MB 2025-02-14 18:54:18,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45602.57 MB 2025-02-14 18:54:18,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 18:54:18,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32356.21 MB 2025-02-14 18:54:18,214 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 18:54:18,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:54:18,216 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:54:18,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:54:18,217 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:54:18,221 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:54:18,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:54:18,222 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:54:18,222 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:54:28,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:54:28,666 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:54:28,671 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:54:28,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:54:28,674 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2564, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:54:28,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:54:28,675 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2564, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:55:08,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:55:08,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:55:08,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.72 seconds 2025-02-14 18:55:08,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:08,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30835.07 MB 2025-02-14 18:55:08,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39909.45 MB 2025-02-14 18:55:08,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9074.38 MB 2025-02-14 18:55:08,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71829.55 MB 2025-02-14 18:55:08,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45061.51 MB 2025-02-14 18:55:08,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26768.05 MB 2025-02-14 18:55:08,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48983.30 MB 2025-02-14 18:55:08,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:55:08,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:55:08,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:55:08,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:08,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39909.45 MB 2025-02-14 18:55:08,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29107.29 MB 2025-02-14 18:55:08,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10802.16 MB 2025-02-14 18:55:08,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45061.51 MB 2025-02-14 18:55:08,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77173.10 MB 2025-02-14 18:55:08,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 32111.59 MB 2025-02-14 18:55:08,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66522.45 MB 2025-02-14 18:55:10,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:55:10,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:55:10,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 18:55:10,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:10,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29107.29 MB 2025-02-14 18:55:10,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29638.13 MB 2025-02-14 18:55:10,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:55:10,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77173.10 MB 2025-02-14 18:55:10,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32864.47 MB 2025-02-14 18:55:10,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -44308.63 MB 2025-02-14 18:55:10,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33617.71 MB 2025-02-14 18:55:10,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:55:10,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:55:10,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:55:10,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:10,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29638.13 MB 2025-02-14 18:55:10,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31527.66 MB 2025-02-14 18:55:10,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:55:10,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32864.47 MB 2025-02-14 18:55:10,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34751.91 MB 2025-02-14 18:55:10,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 18:55:10,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32945.09 MB 2025-02-14 18:55:10,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:55:10,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:55:10,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 18:55:10,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:10,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31527.66 MB 2025-02-14 18:55:10,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33769.52 MB 2025-02-14 18:55:10,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:55:10,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34751.91 MB 2025-02-14 18:55:10,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40886.08 MB 2025-02-14 18:55:10,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:55:10,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39313.80 MB 2025-02-14 18:55:10,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:55:10,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:55:10,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:55:10,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:10,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29638.13 MB 2025-02-14 18:55:10,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33769.52 MB 2025-02-14 18:55:10,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:55:10,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32864.47 MB 2025-02-14 18:55:10,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40886.08 MB 2025-02-14 18:55:10,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 18:55:10,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39313.80 MB 2025-02-14 18:55:10,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:55:10,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:55:10,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:55:10,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:10,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35303.06 MB 2025-02-14 18:55:10,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36070.06 MB 2025-02-14 18:55:10,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:55:10,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40886.08 MB 2025-02-14 18:55:10,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41301.31 MB 2025-02-14 18:55:10,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 18:55:10,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36777.85 MB 2025-02-14 18:55:10,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:55:10,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:55:10,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:55:10,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:10,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36482.95 MB 2025-02-14 18:55:10,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36711.41 MB 2025-02-14 18:55:10,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-14 18:55:10,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41301.31 MB 2025-02-14 18:55:10,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41301.31 MB 2025-02-14 18:55:10,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:55:10,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36914.49 MB 2025-02-14 18:55:10,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:55:10,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:55:10,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.26 seconds 2025-02-14 18:55:10,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:10,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21901.89 MB 2025-02-14 18:55:10,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36912.41 MB 2025-02-14 18:55:10,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15010.52 MB 2025-02-14 18:55:10,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62893.59 MB 2025-02-14 18:55:10,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41301.31 MB 2025-02-14 18:55:10,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21592.28 MB 2025-02-14 18:55:10,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36914.49 MB 2025-02-14 18:55:11,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:55:11,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:55:11,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:55:11,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:11,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36912.41 MB 2025-02-14 18:55:11,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26904.51 MB 2025-02-14 18:55:11,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10007.90 MB 2025-02-14 18:55:11,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41301.31 MB 2025-02-14 18:55:11,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41301.31 MB 2025-02-14 18:55:11,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:55:11,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39423.16 MB 2025-02-14 18:55:11,227 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 18:55:11,227 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:55:11,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:55:11,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:55:11,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:55:11,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:55:11,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26904.51 MB 2025-02-14 18:55:11,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35340.10 MB 2025-02-14 18:55:11,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 18:55:11,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41301.31 MB 2025-02-14 18:55:11,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45495.62 MB 2025-02-14 18:55:11,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 18:55:11,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35340.10 MB 2025-02-14 18:55:11,389 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 18:55:11,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:55:11,391 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:55:11,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:55:11,391 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:55:11,396 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:55:11,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:55:11,397 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:55:11,397 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 18:56:47,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:56:47,324 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:56:47,330 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:56:47,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:56:47,334 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:56:47,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:56:47,335 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:56:50,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:56:50,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:56:50,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.05 seconds 2025-02-14 18:56:50,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:50,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14341.43 MB 2025-02-14 18:56:50,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15038.61 MB 2025-02-14 18:56:50,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 697.17 MB 2025-02-14 18:56:50,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53884.22 MB 2025-02-14 18:56:50,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:56:50,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34351.35 MB 2025-02-14 18:56:50,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24039.30 MB 2025-02-14 18:56:50,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:56:50,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:56:50,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:56:50,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:50,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15038.61 MB 2025-02-14 18:56:50,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15249.97 MB 2025-02-14 18:56:50,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.36 MB 2025-02-14 18:56:50,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:56:50,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:56:50,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:56:50,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17571.40 MB 2025-02-14 18:56:51,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:56:51,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:56:51,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 18:56:51,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15249.97 MB 2025-02-14 18:56:51,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15487.52 MB 2025-02-14 18:56:51,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 237.55 MB 2025-02-14 18:56:51,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:56:51,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:56:51,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:56:51,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19420.66 MB 2025-02-14 18:56:51,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:56:51,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:56:51,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:56:51,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15487.46 MB 2025-02-14 18:56:51,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16332.82 MB 2025-02-14 18:56:51,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 845.36 MB 2025-02-14 18:56:51,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:56:51,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:56:51,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:56:51,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16967.12 MB 2025-02-14 18:56:51,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:56:51,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:56:51,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:56:51,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16332.82 MB 2025-02-14 18:56:51,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17336.08 MB 2025-02-14 18:56:51,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.27 MB 2025-02-14 18:56:51,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:56:51,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21651.00 MB 2025-02-14 18:56:51,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2118.12 MB 2025-02-14 18:56:51,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19817.11 MB 2025-02-14 18:56:51,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:56:51,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:56:51,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:56:51,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15487.46 MB 2025-02-14 18:56:51,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17336.08 MB 2025-02-14 18:56:51,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1848.63 MB 2025-02-14 18:56:51,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:56:51,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21651.00 MB 2025-02-14 18:56:51,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2118.12 MB 2025-02-14 18:56:51,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19817.11 MB 2025-02-14 18:56:51,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:56:51,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:56:51,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:56:51,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18022.34 MB 2025-02-14 18:56:51,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18365.58 MB 2025-02-14 18:56:51,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 343.23 MB 2025-02-14 18:56:51,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21651.00 MB 2025-02-14 18:56:51,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21831.35 MB 2025-02-14 18:56:51,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 18:56:51,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18687.82 MB 2025-02-14 18:56:51,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:56:51,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:56:51,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:56:51,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18550.35 MB 2025-02-14 18:56:51,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18758.91 MB 2025-02-14 18:56:51,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.56 MB 2025-02-14 18:56:51,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21831.35 MB 2025-02-14 18:56:51,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21831.35 MB 2025-02-14 18:56:51,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:56:51,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18790.88 MB 2025-02-14 18:56:51,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:56:51,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:56:51,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.12 seconds 2025-02-14 18:56:51,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13655.07 MB 2025-02-14 18:56:51,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18959.44 MB 2025-02-14 18:56:51,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5304.37 MB 2025-02-14 18:56:51,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53884.22 MB 2025-02-14 18:56:51,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21831.35 MB 2025-02-14 18:56:51,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32052.87 MB 2025-02-14 18:56:51,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18959.44 MB 2025-02-14 18:56:51,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:56:51,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:56:51,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 18:56:51,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18959.44 MB 2025-02-14 18:56:51,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17608.12 MB 2025-02-14 18:56:51,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1351.32 MB 2025-02-14 18:56:51,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21831.35 MB 2025-02-14 18:56:51,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21831.35 MB 2025-02-14 18:56:51,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:56:51,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19059.64 MB 2025-02-14 18:56:51,737 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 18:56:51,738 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1,'] 2025-02-14 18:56:51,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:56:51,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:56:51,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:56:51,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:56:51,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17608.12 MB 2025-02-14 18:56:51,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.72 MB 2025-02-14 18:56:51,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 18:56:51,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21831.35 MB 2025-02-14 18:56:51,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32291.95 MB 2025-02-14 18:56:51,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-14 18:56:51,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26024.72 MB 2025-02-14 18:56:51,905 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 18:56:51,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:56:51,907 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:56:51,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:56:51,908 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:56:51,912 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:56:51,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:56:51,913 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:56:51,914 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1,'] 2025-02-14 18:57:00,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:00,626 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:57:00,631 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:57:00,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:00,634 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2214, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:57:00,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:00,635 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2214, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:57:34,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:57:34,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:57:34,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.19 seconds 2025-02-14 18:57:34,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:34,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28396.21 MB 2025-02-14 18:57:34,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36231.44 MB 2025-02-14 18:57:34,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7835.22 MB 2025-02-14 18:57:34,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40659.58 MB 2025-02-14 18:57:34,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38501.61 MB 2025-02-14 18:57:34,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2157.97 MB 2025-02-14 18:57:34,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45116.15 MB 2025-02-14 18:57:34,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:57:34,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:57:34,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 18:57:34,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:34,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36231.44 MB 2025-02-14 18:57:34,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27288.80 MB 2025-02-14 18:57:34,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8942.64 MB 2025-02-14 18:57:34,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38501.61 MB 2025-02-14 18:57:34,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 72406.27 MB 2025-02-14 18:57:34,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33904.66 MB 2025-02-14 18:57:34,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59325.90 MB 2025-02-14 18:57:36,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:57:36,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:57:36,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 18:57:36,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:36,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27288.80 MB 2025-02-14 18:57:36,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27819.64 MB 2025-02-14 18:57:36,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 18:57:36,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72406.27 MB 2025-02-14 18:57:36,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33497.81 MB 2025-02-14 18:57:36,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38908.46 MB 2025-02-14 18:57:36,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31798.18 MB 2025-02-14 18:57:36,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:57:36,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:57:36,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:57:36,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:36,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27819.64 MB 2025-02-14 18:57:36,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29709.17 MB 2025-02-14 18:57:36,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 18:57:36,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33497.81 MB 2025-02-14 18:57:36,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34441.53 MB 2025-02-14 18:57:36,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 18:57:36,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31126.60 MB 2025-02-14 18:57:37,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:57:37,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:57:37,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 18:57:37,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:37,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29709.17 MB 2025-02-14 18:57:37,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31951.03 MB 2025-02-14 18:57:37,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 18:57:37,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34441.53 MB 2025-02-14 18:57:37,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39631.98 MB 2025-02-14 18:57:37,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 18:57:37,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37495.31 MB 2025-02-14 18:57:37,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:57:37,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:57:37,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 18:57:37,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:37,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27819.64 MB 2025-02-14 18:57:37,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31951.03 MB 2025-02-14 18:57:37,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 18:57:37,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33497.81 MB 2025-02-14 18:57:37,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39631.98 MB 2025-02-14 18:57:37,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 18:57:37,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37495.31 MB 2025-02-14 18:57:37,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:57:37,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:57:37,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 18:57:37,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:37,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33484.57 MB 2025-02-14 18:57:37,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34251.57 MB 2025-02-14 18:57:37,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 18:57:37,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39631.98 MB 2025-02-14 18:57:37,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40049.31 MB 2025-02-14 18:57:37,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 18:57:37,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34959.36 MB 2025-02-14 18:57:37,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:57:37,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:57:37,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:57:37,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:37,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34664.46 MB 2025-02-14 18:57:37,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34893.20 MB 2025-02-14 18:57:37,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-14 18:57:37,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40049.31 MB 2025-02-14 18:57:37,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40049.31 MB 2025-02-14 18:57:37,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:37,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35111.73 MB 2025-02-14 18:57:37,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:57:37,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:57:37,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.72 seconds 2025-02-14 18:57:37,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:37,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20682.46 MB 2025-02-14 18:57:37,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35093.85 MB 2025-02-14 18:57:37,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14411.39 MB 2025-02-14 18:57:37,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40659.58 MB 2025-02-14 18:57:37,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40049.31 MB 2025-02-14 18:57:37,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -610.27 MB 2025-02-14 18:57:37,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35111.73 MB 2025-02-14 18:57:37,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:57:37,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:57:37,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:57:37,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:37,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35093.85 MB 2025-02-14 18:57:37,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25680.37 MB 2025-02-14 18:57:37,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9413.48 MB 2025-02-14 18:57:37,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40049.31 MB 2025-02-14 18:57:37,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40049.31 MB 2025-02-14 18:57:37,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:37,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37600.30 MB 2025-02-14 18:57:37,643 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 18:57:37,644 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:57:37,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:57:37,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:57:37,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:57:37,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:37,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25680.37 MB 2025-02-14 18:57:37,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34102.33 MB 2025-02-14 18:57:37,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 18:57:37,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40049.31 MB 2025-02-14 18:57:37,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48421.14 MB 2025-02-14 18:57:37,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 18:57:37,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34102.33 MB 2025-02-14 18:57:37,810 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 18:57:37,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:37,811 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:57:37,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:37,812 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:57:37,817 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:57:37,818 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:37,818 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:57:37,818 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 18:57:48,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:48,411 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:57:48,419 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:57:48,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:48,425 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 156, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:57:48,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:48,427 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 156, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:57:50,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:57:50,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:57:50,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-14 18:57:50,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:50,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14055.74 MB 2025-02-14 18:57:50,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14607.81 MB 2025-02-14 18:57:50,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.08 MB 2025-02-14 18:57:50,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56792.97 MB 2025-02-14 18:57:50,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23481.81 MB 2025-02-14 18:57:50,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33311.16 MB 2025-02-14 18:57:50,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23527.11 MB 2025-02-14 18:57:50,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:57:50,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:57:50,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:57:50,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:50,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14607.81 MB 2025-02-14 18:57:50,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14875.29 MB 2025-02-14 18:57:50,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.48 MB 2025-02-14 18:57:50,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23481.81 MB 2025-02-14 18:57:50,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23481.81 MB 2025-02-14 18:57:50,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:50,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16841.53 MB 2025-02-14 18:57:51,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:57:51,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:57:51,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.76 seconds 2025-02-14 18:57:51,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:51,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14875.29 MB 2025-02-14 18:57:51,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.32 MB 2025-02-14 18:57:51,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.03 MB 2025-02-14 18:57:51,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23481.81 MB 2025-02-14 18:57:51,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23481.81 MB 2025-02-14 18:57:51,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:51,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.94 MB 2025-02-14 18:57:51,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:57:51,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:57:51,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 18:57:51,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:51,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.26 MB 2025-02-14 18:57:51,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15819.00 MB 2025-02-14 18:57:51,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 736.74 MB 2025-02-14 18:57:51,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23481.81 MB 2025-02-14 18:57:51,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23481.81 MB 2025-02-14 18:57:51,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:51,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16371.80 MB 2025-02-14 18:57:51,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:57:51,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:57:51,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 18:57:51,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:51,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15819.00 MB 2025-02-14 18:57:51,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16693.36 MB 2025-02-14 18:57:51,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 874.36 MB 2025-02-14 18:57:51,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23481.81 MB 2025-02-14 18:57:51,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23481.81 MB 2025-02-14 18:57:51,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:51,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18855.59 MB 2025-02-14 18:57:51,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:57:51,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:57:51,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 18:57:51,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:51,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.26 MB 2025-02-14 18:57:51,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16693.36 MB 2025-02-14 18:57:51,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1611.10 MB 2025-02-14 18:57:51,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23481.81 MB 2025-02-14 18:57:51,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23481.81 MB 2025-02-14 18:57:51,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:51,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18855.59 MB 2025-02-14 18:57:51,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:57:51,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:57:51,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 18:57:51,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:51,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17291.44 MB 2025-02-14 18:57:51,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17590.57 MB 2025-02-14 18:57:51,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.13 MB 2025-02-14 18:57:51,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23481.81 MB 2025-02-14 18:57:51,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23639.10 MB 2025-02-14 18:57:51,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 157.29 MB 2025-02-14 18:57:51,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.87 MB 2025-02-14 18:57:51,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:57:51,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:57:51,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:57:51,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:51,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17751.61 MB 2025-02-14 18:57:51,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17962.83 MB 2025-02-14 18:57:51,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.22 MB 2025-02-14 18:57:51,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23639.10 MB 2025-02-14 18:57:51,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23639.10 MB 2025-02-14 18:57:51,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:51,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17976.93 MB 2025-02-14 18:57:51,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:57:51,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:57:51,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-14 18:57:51,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:51,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13512.22 MB 2025-02-14 18:57:51,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18163.88 MB 2025-02-14 18:57:51,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4651.65 MB 2025-02-14 18:57:51,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56792.97 MB 2025-02-14 18:57:51,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23639.10 MB 2025-02-14 18:57:51,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33153.88 MB 2025-02-14 18:57:51,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18163.88 MB 2025-02-14 18:57:52,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:57:52,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:57:52,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:57:52,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:52,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18163.88 MB 2025-02-14 18:57:52,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17364.72 MB 2025-02-14 18:57:52,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -799.15 MB 2025-02-14 18:57:52,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23639.10 MB 2025-02-14 18:57:52,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23639.10 MB 2025-02-14 18:57:52,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:57:52,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19067.97 MB 2025-02-14 18:57:52,179 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 18:57:52,179 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:57:52,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:57:52,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:57:52,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:57:52,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:57:52,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17364.72 MB 2025-02-14 18:57:52,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25803.56 MB 2025-02-14 18:57:52,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 18:57:52,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23639.10 MB 2025-02-14 18:57:52,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32027.71 MB 2025-02-14 18:57:52,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 18:57:52,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25803.56 MB 2025-02-14 18:57:52,342 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 18:57:52,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:52,343 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:57:52,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:52,344 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:57:52,348 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:57:52,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:57:52,350 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:57:52,350 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:58:06,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:06,346 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:58:06,351 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:58:06,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:06,355 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 185, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:58:06,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:06,356 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 185, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:58:09,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:58:09,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:58:09,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.89 seconds 2025-02-14 18:58:09,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:09,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14257.82 MB 2025-02-14 18:58:09,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14912.52 MB 2025-02-14 18:58:09,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 654.70 MB 2025-02-14 18:58:09,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40416.31 MB 2025-02-14 18:58:09,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20931.67 MB 2025-02-14 18:58:09,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19484.64 MB 2025-02-14 18:58:09,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23729.19 MB 2025-02-14 18:58:09,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:58:09,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:58:09,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:58:09,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:09,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14912.52 MB 2025-02-14 18:58:09,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15201.63 MB 2025-02-14 18:58:09,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 289.11 MB 2025-02-14 18:58:09,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20931.67 MB 2025-02-14 18:58:09,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20931.67 MB 2025-02-14 18:58:09,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:09,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17486.77 MB 2025-02-14 18:58:10,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:58:10,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:58:10,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 18:58:10,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15201.63 MB 2025-02-14 18:58:10,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15441.84 MB 2025-02-14 18:58:10,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.21 MB 2025-02-14 18:58:10,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20931.67 MB 2025-02-14 18:58:10,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20931.67 MB 2025-02-14 18:58:10,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:10,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19371.28 MB 2025-02-14 18:58:10,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:58:10,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:58:10,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:58:10,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15441.77 MB 2025-02-14 18:58:10,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16296.58 MB 2025-02-14 18:58:10,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 854.81 MB 2025-02-14 18:58:10,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20931.67 MB 2025-02-14 18:58:10,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20931.67 MB 2025-02-14 18:58:10,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:10,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16937.97 MB 2025-02-14 18:58:10,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:58:10,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:58:10,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:58:10,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16296.58 MB 2025-02-14 18:58:10,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17311.05 MB 2025-02-14 18:58:10,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1014.48 MB 2025-02-14 18:58:10,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20931.67 MB 2025-02-14 18:58:10,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20931.67 MB 2025-02-14 18:58:10,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:10,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19819.81 MB 2025-02-14 18:58:10,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:58:10,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:58:10,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:58:10,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15441.77 MB 2025-02-14 18:58:10,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17311.05 MB 2025-02-14 18:58:10,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1869.28 MB 2025-02-14 18:58:10,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20931.67 MB 2025-02-14 18:58:10,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20931.67 MB 2025-02-14 18:58:10,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:10,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19819.81 MB 2025-02-14 18:58:10,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:58:10,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:58:10,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 18:58:10,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18004.98 MB 2025-02-14 18:58:10,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18352.05 MB 2025-02-14 18:58:10,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.07 MB 2025-02-14 18:58:10,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20931.67 MB 2025-02-14 18:58:10,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 18:58:10,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-14 18:58:10,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18676.90 MB 2025-02-14 18:58:10,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:58:10,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:58:10,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:58:10,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18538.89 MB 2025-02-14 18:58:10,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18754.84 MB 2025-02-14 18:58:10,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 215.95 MB 2025-02-14 18:58:10,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21118.32 MB 2025-02-14 18:58:10,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 18:58:10,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:10,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18794.27 MB 2025-02-14 18:58:10,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:58:10,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:58:10,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.97 seconds 2025-02-14 18:58:10,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13613.26 MB 2025-02-14 18:58:10,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18955.40 MB 2025-02-14 18:58:10,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5342.14 MB 2025-02-14 18:58:10,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40416.31 MB 2025-02-14 18:58:10,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 18:58:10,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19297.99 MB 2025-02-14 18:58:10,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18955.40 MB 2025-02-14 18:58:10,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:58:10,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:58:10,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:58:10,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18955.40 MB 2025-02-14 18:58:10,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17576.13 MB 2025-02-14 18:58:10,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1379.27 MB 2025-02-14 18:58:10,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21118.32 MB 2025-02-14 18:58:10,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 18:58:10,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:10,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18955.41 MB 2025-02-14 18:58:10,612 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 18:58:10,612 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 18:58:10,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:58:10,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:58:10,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:58:10,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:10,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17576.13 MB 2025-02-14 18:58:10,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25993.87 MB 2025-02-14 18:58:10,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-14 18:58:10,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21118.32 MB 2025-02-14 18:58:10,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31578.91 MB 2025-02-14 18:58:10,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-14 18:58:10,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25993.87 MB 2025-02-14 18:58:10,776 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 18:58:10,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:10,778 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:58:10,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:10,779 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:58:10,783 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:58:10,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:10,784 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:58:10,784 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 18:58:52,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:52,687 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 18:58:52,692 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 18:58:52,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:52,695 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 274, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 18:58:52,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:52,696 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 274, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 18:58:56,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 18:58:56,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 18:58:56,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.21 seconds 2025-02-14 18:58:56,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:56,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.98 MB 2025-02-14 18:58:56,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15847.65 MB 2025-02-14 18:58:56,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 969.67 MB 2025-02-14 18:58:56,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39946.55 MB 2025-02-14 18:58:56,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19295.90 MB 2025-02-14 18:58:56,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20650.66 MB 2025-02-14 18:58:56,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24802.34 MB 2025-02-14 18:58:56,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 18:58:56,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 18:58:56,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:58:56,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:56,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15847.65 MB 2025-02-14 18:58:56,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16269.15 MB 2025-02-14 18:58:56,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 421.49 MB 2025-02-14 18:58:56,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19295.90 MB 2025-02-14 18:58:56,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21187.53 MB 2025-02-14 18:58:56,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1891.63 MB 2025-02-14 18:58:56,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19606.86 MB 2025-02-14 18:58:58,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 18:58:58,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 18:58:58,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.29 seconds 2025-02-14 18:58:58,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16269.15 MB 2025-02-14 18:58:58,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16623.48 MB 2025-02-14 18:58:58,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 354.34 MB 2025-02-14 18:58:58,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21187.53 MB 2025-02-14 18:58:58,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 18:58:58,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1654.65 MB 2025-02-14 18:58:58,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20609.70 MB 2025-02-14 18:58:58,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 18:58:58,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 18:58:58,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:58:58,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16623.48 MB 2025-02-14 18:58:58,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17884.49 MB 2025-02-14 18:58:58,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.00 MB 2025-02-14 18:58:58,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:58:58,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20795.36 MB 2025-02-14 18:58:58,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1262.49 MB 2025-02-14 18:58:58,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18830.62 MB 2025-02-14 18:58:58,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 18:58:58,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 18:58:58,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 18:58:58,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17884.49 MB 2025-02-14 18:58:58,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19380.95 MB 2025-02-14 18:58:58,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1496.46 MB 2025-02-14 18:58:58,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20795.36 MB 2025-02-14 18:58:58,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24584.91 MB 2025-02-14 18:58:58,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3789.55 MB 2025-02-14 18:58:58,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23081.73 MB 2025-02-14 18:58:58,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 18:58:58,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 18:58:58,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 18:58:58,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16623.48 MB 2025-02-14 18:58:58,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19380.95 MB 2025-02-14 18:58:58,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2757.46 MB 2025-02-14 18:58:58,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 18:58:58,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24584.91 MB 2025-02-14 18:58:58,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5052.04 MB 2025-02-14 18:58:58,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23081.73 MB 2025-02-14 18:58:58,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 18:58:58,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 18:58:58,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 18:58:58,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20404.59 MB 2025-02-14 18:58:58,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20916.56 MB 2025-02-14 18:58:58,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 511.97 MB 2025-02-14 18:58:58,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24584.91 MB 2025-02-14 18:58:58,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24859.64 MB 2025-02-14 18:58:58,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 274.73 MB 2025-02-14 18:58:58,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21389.01 MB 2025-02-14 18:58:58,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 18:58:58,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 18:58:58,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 18:58:58,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21192.17 MB 2025-02-14 18:58:58,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21401.15 MB 2025-02-14 18:58:58,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.99 MB 2025-02-14 18:58:58,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24859.64 MB 2025-02-14 18:58:58,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24859.64 MB 2025-02-14 18:58:58,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 18:58:58,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21503.49 MB 2025-02-14 18:58:58,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 18:58:58,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 18:58:58,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.79 seconds 2025-02-14 18:58:58,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13923.34 MB 2025-02-14 18:58:58,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21602.23 MB 2025-02-14 18:58:58,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7678.88 MB 2025-02-14 18:58:58,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39946.55 MB 2025-02-14 18:58:58,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24859.64 MB 2025-02-14 18:58:58,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15086.91 MB 2025-02-14 18:58:58,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21602.23 MB 2025-02-14 18:58:58,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 18:58:58,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 18:58:58,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 18:58:58,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21602.23 MB 2025-02-14 18:58:58,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24616.26 MB 2025-02-14 18:58:58,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 18:58:58,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24859.64 MB 2025-02-14 18:58:58,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26067.60 MB 2025-02-14 18:58:58,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1207.96 MB 2025-02-14 18:58:58,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24917.89 MB 2025-02-14 18:58:58,778 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 18:58:58,778 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 18:58:58,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 18:58:58,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 18:58:58,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 18:58:58,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 18:58:58,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18299.68 MB 2025-02-14 18:58:58,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26738.70 MB 2025-02-14 18:58:58,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 18:58:58,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26067.60 MB 2025-02-14 18:58:58,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36557.55 MB 2025-02-14 18:58:58,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 18:58:58,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26738.70 MB 2025-02-14 18:58:58,944 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 18:58:58,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:58,945 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 18:58:58,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:58,946 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 18:58:58,951 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 18:58:58,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 18:58:58,952 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 18:58:58,952 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:00:09,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:00:09,236 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:00:09,244 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:00:09,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:00:09,251 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 932, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:00:09,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:00:09,253 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 932, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:00:23,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:00:23,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:00:23,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.27 seconds 2025-02-14 19:00:23,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:23,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19463.03 MB 2025-02-14 19:00:23,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22761.85 MB 2025-02-14 19:00:23,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3298.82 MB 2025-02-14 19:00:23,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49142.56 MB 2025-02-14 19:00:23,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 19:00:23,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21434.99 MB 2025-02-14 19:00:23,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31652.31 MB 2025-02-14 19:00:23,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:00:23,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:00:23,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:00:23,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:23,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22761.85 MB 2025-02-14 19:00:23,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20624.08 MB 2025-02-14 19:00:23,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2137.78 MB 2025-02-14 19:00:23,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 19:00:23,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37144.76 MB 2025-02-14 19:00:23,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 19:00:23,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33042.43 MB 2025-02-14 19:00:25,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:00:25,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:00:25,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 19:00:25,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:25,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20624.08 MB 2025-02-14 19:00:25,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21154.92 MB 2025-02-14 19:00:25,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:00:25,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37144.76 MB 2025-02-14 19:00:25,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26533.17 MB 2025-02-14 19:00:25,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10611.59 MB 2025-02-14 19:00:25,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25133.47 MB 2025-02-14 19:00:25,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:00:25,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:00:25,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:00:25,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:25,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21154.92 MB 2025-02-14 19:00:25,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23044.45 MB 2025-02-14 19:00:25,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:00:25,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26533.17 MB 2025-02-14 19:00:25,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26533.17 MB 2025-02-14 19:00:25,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:00:25,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24461.88 MB 2025-02-14 19:00:25,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:00:25,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:00:25,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:00:25,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:25,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23044.45 MB 2025-02-14 19:00:25,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25286.31 MB 2025-02-14 19:00:25,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:00:25,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26533.17 MB 2025-02-14 19:00:25,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32667.34 MB 2025-02-14 19:00:25,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:00:25,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30830.59 MB 2025-02-14 19:00:25,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:00:25,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:00:25,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:00:25,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:25,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21154.92 MB 2025-02-14 19:00:25,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25286.31 MB 2025-02-14 19:00:25,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:00:25,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26533.17 MB 2025-02-14 19:00:25,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32667.34 MB 2025-02-14 19:00:25,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:00:25,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30830.59 MB 2025-02-14 19:00:25,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:00:25,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:00:25,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:00:25,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:25,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26819.85 MB 2025-02-14 19:00:25,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27586.85 MB 2025-02-14 19:00:25,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:00:25,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32667.34 MB 2025-02-14 19:00:25,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 19:00:25,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:00:25,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28294.64 MB 2025-02-14 19:00:25,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:00:25,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:00:25,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:00:25,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:25,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27999.74 MB 2025-02-14 19:00:25,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28228.70 MB 2025-02-14 19:00:25,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 19:00:25,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 19:00:25,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 19:00:25,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:00:25,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.04 MB 2025-02-14 19:00:25,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:00:25,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:00:25,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.64 seconds 2025-02-14 19:00:25,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:25,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16215.87 MB 2025-02-14 19:00:25,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28429.58 MB 2025-02-14 19:00:25,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12213.71 MB 2025-02-14 19:00:25,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49142.56 MB 2025-02-14 19:00:25,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 19:00:25,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16059.99 MB 2025-02-14 19:00:25,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.04 MB 2025-02-14 19:00:26,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:00:26,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:00:26,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:00:26,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:26,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28429.58 MB 2025-02-14 19:00:26,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21217.21 MB 2025-02-14 19:00:26,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7212.37 MB 2025-02-14 19:00:26,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 19:00:26,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 19:00:26,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:00:26,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30938.79 MB 2025-02-14 19:00:26,188 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 19:00:26,188 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:00:26,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:00:26,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:00:26,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:00:26,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:00:26,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21217.21 MB 2025-02-14 19:00:26,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29647.89 MB 2025-02-14 19:00:26,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 19:00:26,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 19:00:26,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41464.89 MB 2025-02-14 19:00:26,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 19:00:26,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29647.89 MB 2025-02-14 19:00:26,355 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 19:00:26,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:00:26,356 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:00:26,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:00:26,357 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:00:26,362 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:00:26,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:00:26,363 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:00:26,363 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:01:34,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:01:34,262 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:01:34,267 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:01:34,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:01:34,270 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1487, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:01:34,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:01:34,271 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1487, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:01:57,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:01:57,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:01:57,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.78 seconds 2025-02-14 19:01:57,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:57,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23330.36 MB 2025-02-14 19:01:57,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28592.77 MB 2025-02-14 19:01:57,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5262.41 MB 2025-02-14 19:01:57,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54037.32 MB 2025-02-14 19:01:57,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-14 19:01:57,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18081.64 MB 2025-02-14 19:01:57,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37558.07 MB 2025-02-14 19:01:57,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:01:57,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:01:57,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:01:57,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:57,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28592.77 MB 2025-02-14 19:01:57,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.30 MB 2025-02-14 19:01:57,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5084.47 MB 2025-02-14 19:01:57,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35955.67 MB 2025-02-14 19:01:57,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47034.93 MB 2025-02-14 19:01:57,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11079.25 MB 2025-02-14 19:01:57,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42275.86 MB 2025-02-14 19:01:59,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:01:59,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:01:59,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 19:01:59,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.30 MB 2025-02-14 19:01:59,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24039.14 MB 2025-02-14 19:01:59,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:01:59,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47034.93 MB 2025-02-14 19:01:59,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30691.82 MB 2025-02-14 19:01:59,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16343.11 MB 2025-02-14 19:01:59,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28017.69 MB 2025-02-14 19:01:59,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:01:59,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:01:59,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:01:59,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24039.14 MB 2025-02-14 19:01:59,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25928.68 MB 2025-02-14 19:01:59,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:01:59,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30691.82 MB 2025-02-14 19:01:59,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30691.82 MB 2025-02-14 19:01:59,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:01:59,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27346.10 MB 2025-02-14 19:01:59,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:01:59,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:01:59,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:01:59,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25928.68 MB 2025-02-14 19:01:59,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28170.53 MB 2025-02-14 19:01:59,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:01:59,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30691.82 MB 2025-02-14 19:01:59,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35882.27 MB 2025-02-14 19:01:59,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 19:01:59,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33714.81 MB 2025-02-14 19:01:59,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:01:59,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:01:59,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:01:59,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24039.14 MB 2025-02-14 19:01:59,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28170.53 MB 2025-02-14 19:01:59,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:01:59,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30691.82 MB 2025-02-14 19:01:59,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35882.27 MB 2025-02-14 19:01:59,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 19:01:59,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33714.81 MB 2025-02-14 19:01:59,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:01:59,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:01:59,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:01:59,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29704.07 MB 2025-02-14 19:01:59,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30471.08 MB 2025-02-14 19:01:59,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:01:59,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35882.27 MB 2025-02-14 19:01:59,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36295.41 MB 2025-02-14 19:01:59,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 19:01:59,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31178.86 MB 2025-02-14 19:01:59,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:01:59,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:01:59,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:01:59,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30883.96 MB 2025-02-14 19:01:59,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31111.82 MB 2025-02-14 19:01:59,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.85 MB 2025-02-14 19:01:59,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36295.41 MB 2025-02-14 19:01:59,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36295.41 MB 2025-02-14 19:01:59,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:01:59,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31315.53 MB 2025-02-14 19:01:59,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:01:59,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:01:59,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.18 seconds 2025-02-14 19:01:59,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18149.53 MB 2025-02-14 19:01:59,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31312.28 MB 2025-02-14 19:01:59,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13162.74 MB 2025-02-14 19:01:59,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54037.32 MB 2025-02-14 19:01:59,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36295.41 MB 2025-02-14 19:01:59,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17741.91 MB 2025-02-14 19:01:59,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31315.53 MB 2025-02-14 19:01:59,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:01:59,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:01:59,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:01:59,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31312.28 MB 2025-02-14 19:01:59,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23144.42 MB 2025-02-14 19:01:59,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8167.85 MB 2025-02-14 19:01:59,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36295.41 MB 2025-02-14 19:01:59,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36295.41 MB 2025-02-14 19:01:59,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:01:59,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33816.29 MB 2025-02-14 19:01:59,736 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 19:01:59,736 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:01:59,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:01:59,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:01:59,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:01:59,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:01:59,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23144.42 MB 2025-02-14 19:01:59,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31557.95 MB 2025-02-14 19:01:59,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 19:01:59,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36295.41 MB 2025-02-14 19:01:59,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44658.85 MB 2025-02-14 19:01:59,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 19:01:59,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31557.95 MB 2025-02-14 19:01:59,899 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 19:01:59,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:01:59,900 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:01:59,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:01:59,901 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:01:59,906 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:01:59,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:01:59,907 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:01:59,907 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:03:38,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:03:38,637 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:03:38,642 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:03:38,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:03:38,646 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:03:38,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:03:38,647 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:04:09,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:04:09,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:04:09,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.79 seconds 2025-02-14 19:04:09,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:09,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26932.90 MB 2025-02-14 19:04:09,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34025.47 MB 2025-02-14 19:04:09,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7092.57 MB 2025-02-14 19:04:09,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53022.29 MB 2025-02-14 19:04:09,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37752.93 MB 2025-02-14 19:04:09,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15269.36 MB 2025-02-14 19:04:09,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42973.36 MB 2025-02-14 19:04:09,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:04:09,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:04:09,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:04:09,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:09,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34025.47 MB 2025-02-14 19:04:09,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24426.22 MB 2025-02-14 19:04:09,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9599.25 MB 2025-02-14 19:04:09,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37752.93 MB 2025-02-14 19:04:09,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37752.93 MB 2025-02-14 19:04:09,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:04:09,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36118.44 MB 2025-02-14 19:04:10,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:04:10,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:04:10,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 19:04:10,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24426.22 MB 2025-02-14 19:04:10,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24622.63 MB 2025-02-14 19:04:10,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-14 19:04:10,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37752.93 MB 2025-02-14 19:04:10,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 19:04:10,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7092.57 MB 2025-02-14 19:04:10,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28595.87 MB 2025-02-14 19:04:10,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:04:10,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:04:10,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:04:10,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24622.63 MB 2025-02-14 19:04:10,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25321.59 MB 2025-02-14 19:04:10,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-14 19:04:10,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 19:04:10,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 19:04:10,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:04:10,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25846.04 MB 2025-02-14 19:04:10,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:04:10,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:04:10,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:04:10,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25321.59 MB 2025-02-14 19:04:10,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26151.12 MB 2025-02-14 19:04:10,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 829.53 MB 2025-02-14 19:04:10,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 19:04:10,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 19:04:10,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:04:10,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28202.46 MB 2025-02-14 19:04:10,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:04:10,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:04:10,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:04:10,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24622.63 MB 2025-02-14 19:04:10,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26151.12 MB 2025-02-14 19:04:10,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1528.49 MB 2025-02-14 19:04:10,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 19:04:10,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 19:04:10,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:04:10,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28202.46 MB 2025-02-14 19:04:10,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:04:10,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:04:10,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:04:10,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26718.53 MB 2025-02-14 19:04:10,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27002.32 MB 2025-02-14 19:04:10,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.79 MB 2025-02-14 19:04:10,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 19:04:10,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30811.36 MB 2025-02-14 19:04:10,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 19:04:10,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27273.22 MB 2025-02-14 19:04:10,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:04:10,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:04:10,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:04:10,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27155.09 MB 2025-02-14 19:04:10,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27250.03 MB 2025-02-14 19:04:10,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 94.94 MB 2025-02-14 19:04:10,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30811.36 MB 2025-02-14 19:04:10,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30813.45 MB 2025-02-14 19:04:10,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 19:04:10,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27250.03 MB 2025-02-14 19:04:10,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:04:10,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:04:10,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.74 seconds 2025-02-14 19:04:10,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-14 19:04:10,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27333.37 MB 2025-02-14 19:04:10,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7382.57 MB 2025-02-14 19:04:10,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53022.29 MB 2025-02-14 19:04:10,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30813.45 MB 2025-02-14 19:04:10,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22208.84 MB 2025-02-14 19:04:10,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27333.37 MB 2025-02-14 19:04:10,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:04:10,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:04:10,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:04:10,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27333.37 MB 2025-02-14 19:04:10,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21942.32 MB 2025-02-14 19:04:10,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5391.05 MB 2025-02-14 19:04:10,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30813.45 MB 2025-02-14 19:04:10,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30813.45 MB 2025-02-14 19:04:10,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:04:10,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27791.46 MB 2025-02-14 19:04:10,501 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 3375, cut from 3377 2025-02-14 19:04:10,502 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 1 ('] 2025-02-14 19:04:10,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:04:10,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:04:10,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:04:10,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:04:10,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21942.32 MB 2025-02-14 19:04:10,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25439.91 MB 2025-02-14 19:04:10,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3497.59 MB 2025-02-14 19:04:10,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30813.45 MB 2025-02-14 19:04:10,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30813.45 MB 2025-02-14 19:04:10,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:04:10,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25439.91 MB 2025-02-14 19:04:10,569 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 3167] 2025-02-14 19:04:10,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:04:10,571 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:04:10,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:04:10,572 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:04:10,576 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:04:10,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:04:10,578 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:04:10,578 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 1 ('] 2025-02-14 19:05:11,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:11,293 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:05:11,298 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:05:11,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:11,301 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2064, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:05:11,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:11,302 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2064, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:05:43,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:05:43,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:05:43,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.96 seconds 2025-02-14 19:05:43,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:43,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27352.03 MB 2025-02-14 19:05:43,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34656.41 MB 2025-02-14 19:05:43,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7304.38 MB 2025-02-14 19:05:43,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36091.99 MB 2025-02-14 19:05:43,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37323.01 MB 2025-02-14 19:05:43,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1231.03 MB 2025-02-14 19:05:43,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43618.98 MB 2025-02-14 19:05:43,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:05:43,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:05:43,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 19:05:43,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:43,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34656.41 MB 2025-02-14 19:05:43,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26510.03 MB 2025-02-14 19:05:43,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8146.38 MB 2025-02-14 19:05:43,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37323.01 MB 2025-02-14 19:05:43,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66584.58 MB 2025-02-14 19:05:43,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29261.56 MB 2025-02-14 19:05:43,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54610.26 MB 2025-02-14 19:05:45,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:05:45,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:05:45,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 19:05:45,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:45,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26510.03 MB 2025-02-14 19:05:45,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27040.87 MB 2025-02-14 19:05:45,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:05:45,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66584.58 MB 2025-02-14 19:05:45,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32851.89 MB 2025-02-14 19:05:45,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33732.69 MB 2025-02-14 19:05:45,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31019.42 MB 2025-02-14 19:05:45,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:05:45,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:05:45,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:05:45,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:45,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27040.87 MB 2025-02-14 19:05:45,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28930.41 MB 2025-02-14 19:05:45,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:05:45,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32851.89 MB 2025-02-14 19:05:45,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32851.89 MB 2025-02-14 19:05:45,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:05:45,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30347.84 MB 2025-02-14 19:05:45,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:05:45,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:05:45,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:05:45,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:45,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28930.41 MB 2025-02-14 19:05:45,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31172.26 MB 2025-02-14 19:05:45,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:05:45,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32851.89 MB 2025-02-14 19:05:45,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38514.20 MB 2025-02-14 19:05:45,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:05:45,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36716.55 MB 2025-02-14 19:05:45,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:05:45,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:05:45,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 19:05:45,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:45,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27040.87 MB 2025-02-14 19:05:45,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31172.26 MB 2025-02-14 19:05:45,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:05:45,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32851.89 MB 2025-02-14 19:05:45,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38514.20 MB 2025-02-14 19:05:45,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:05:45,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36716.55 MB 2025-02-14 19:05:45,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:05:45,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:05:45,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:05:45,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:45,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32705.81 MB 2025-02-14 19:05:45,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33472.81 MB 2025-02-14 19:05:45,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:05:45,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38514.20 MB 2025-02-14 19:05:45,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38929.43 MB 2025-02-14 19:05:45,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:05:45,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34180.60 MB 2025-02-14 19:05:45,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:05:45,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:05:45,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:05:45,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:45,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33885.70 MB 2025-02-14 19:05:45,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34114.04 MB 2025-02-14 19:05:45,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 19:05:45,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38929.43 MB 2025-02-14 19:05:45,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38929.43 MB 2025-02-14 19:05:45,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:05:45,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34321.59 MB 2025-02-14 19:05:45,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:05:45,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:05:45,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.46 seconds 2025-02-14 19:05:45,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:45,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20160.89 MB 2025-02-14 19:05:45,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34314.53 MB 2025-02-14 19:05:45,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14153.64 MB 2025-02-14 19:05:45,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36091.99 MB 2025-02-14 19:05:45,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38929.43 MB 2025-02-14 19:05:45,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2837.45 MB 2025-02-14 19:05:45,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34321.59 MB 2025-02-14 19:05:46,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:05:46,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:05:46,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:05:46,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:46,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34314.53 MB 2025-02-14 19:05:46,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25152.93 MB 2025-02-14 19:05:46,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9161.60 MB 2025-02-14 19:05:46,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38929.43 MB 2025-02-14 19:05:46,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38929.43 MB 2025-02-14 19:05:46,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:05:46,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36816.06 MB 2025-02-14 19:05:46,047 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 19:05:46,047 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:05:46,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:05:46,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:05:46,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:05:46,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:05:46,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25152.93 MB 2025-02-14 19:05:46,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33558.04 MB 2025-02-14 19:05:46,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 19:05:46,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38929.43 MB 2025-02-14 19:05:46,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43106.96 MB 2025-02-14 19:05:46,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 19:05:46,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33558.04 MB 2025-02-14 19:05:46,208 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 19:05:46,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:46,210 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:05:46,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:46,211 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:05:46,215 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:05:46,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:46,216 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:05:46,217 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:05:56,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:56,335 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:05:56,343 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:05:56,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:56,349 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1353, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:05:56,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:05:56,351 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1353, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:06:17,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:06:17,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:06:17,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.09 seconds 2025-02-14 19:06:17,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:17,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22396.63 MB 2025-02-14 19:06:17,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27184.82 MB 2025-02-14 19:06:17,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4788.19 MB 2025-02-14 19:06:17,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55641.64 MB 2025-02-14 19:06:17,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34802.24 MB 2025-02-14 19:06:17,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20839.40 MB 2025-02-14 19:06:17,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36171.35 MB 2025-02-14 19:06:17,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:06:17,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:06:17,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:06:17,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:17,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27184.82 MB 2025-02-14 19:06:17,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22811.68 MB 2025-02-14 19:06:17,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4373.14 MB 2025-02-14 19:06:17,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34802.24 MB 2025-02-14 19:06:17,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45696.94 MB 2025-02-14 19:06:17,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10894.70 MB 2025-02-14 19:06:17,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40949.20 MB 2025-02-14 19:06:19,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:06:19,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:06:19,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 19:06:19,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:19,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22811.68 MB 2025-02-14 19:06:19,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23342.52 MB 2025-02-14 19:06:19,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:06:19,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45696.94 MB 2025-02-14 19:06:19,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30012.34 MB 2025-02-14 19:06:19,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15684.60 MB 2025-02-14 19:06:19,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27321.06 MB 2025-02-14 19:06:19,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:06:19,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:06:19,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:06:19,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:19,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23342.52 MB 2025-02-14 19:06:19,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25232.05 MB 2025-02-14 19:06:19,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:06:19,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30012.34 MB 2025-02-14 19:06:19,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30012.34 MB 2025-02-14 19:06:19,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:19,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26649.48 MB 2025-02-14 19:06:19,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:06:19,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:06:19,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:06:19,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:19,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25232.05 MB 2025-02-14 19:06:19,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27473.91 MB 2025-02-14 19:06:19,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:06:19,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30012.34 MB 2025-02-14 19:06:19,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35674.65 MB 2025-02-14 19:06:19,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:06:19,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33018.19 MB 2025-02-14 19:06:19,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:06:19,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:06:19,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:06:19,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:19,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23342.52 MB 2025-02-14 19:06:19,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27473.91 MB 2025-02-14 19:06:19,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:06:19,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30012.34 MB 2025-02-14 19:06:19,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35674.65 MB 2025-02-14 19:06:19,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:06:19,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33018.19 MB 2025-02-14 19:06:19,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:06:19,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:06:19,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:06:19,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:19,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29007.45 MB 2025-02-14 19:06:19,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29774.45 MB 2025-02-14 19:06:19,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:06:19,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35674.65 MB 2025-02-14 19:06:19,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36087.79 MB 2025-02-14 19:06:19,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 19:06:19,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30482.24 MB 2025-02-14 19:06:19,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:06:19,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:06:19,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:06:19,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:19,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30187.34 MB 2025-02-14 19:06:19,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30416.88 MB 2025-02-14 19:06:19,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.54 MB 2025-02-14 19:06:19,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36087.79 MB 2025-02-14 19:06:19,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36087.79 MB 2025-02-14 19:06:19,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:19,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30622.16 MB 2025-02-14 19:06:19,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:06:19,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:06:19,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.51 seconds 2025-02-14 19:06:19,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:19,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17682.67 MB 2025-02-14 19:06:19,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30617.95 MB 2025-02-14 19:06:19,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12935.29 MB 2025-02-14 19:06:19,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55641.64 MB 2025-02-14 19:06:19,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36087.79 MB 2025-02-14 19:06:19,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19553.85 MB 2025-02-14 19:06:19,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30622.16 MB 2025-02-14 19:06:20,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:06:20,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:06:20,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:06:20,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:20,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30617.95 MB 2025-02-14 19:06:20,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22687.06 MB 2025-02-14 19:06:20,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7930.90 MB 2025-02-14 19:06:20,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36087.79 MB 2025-02-14 19:06:20,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36087.79 MB 2025-02-14 19:06:20,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:20,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33129.62 MB 2025-02-14 19:06:20,146 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:06:20,146 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:06:20,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:06:20,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:06:20,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:06:20,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:20,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22687.06 MB 2025-02-14 19:06:20,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31126.08 MB 2025-02-14 19:06:20,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:06:20,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36087.79 MB 2025-02-14 19:06:20,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44478.50 MB 2025-02-14 19:06:20,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:06:20,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31126.08 MB 2025-02-14 19:06:20,310 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:06:20,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:20,311 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:06:20,312 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:20,312 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:06:20,317 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:06:20,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:20,318 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:06:20,318 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:06:32,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:32,704 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:06:32,712 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:06:32,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:32,719 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 149, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:06:32,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:32,721 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 149, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:06:35,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:06:35,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:06:35,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.41 seconds 2025-02-14 19:06:35,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:35,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14006.96 MB 2025-02-14 19:06:35,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14534.26 MB 2025-02-14 19:06:35,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 527.30 MB 2025-02-14 19:06:35,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57063.51 MB 2025-02-14 19:06:35,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21655.19 MB 2025-02-14 19:06:35,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35408.31 MB 2025-02-14 19:06:35,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23478.33 MB 2025-02-14 19:06:35,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:06:35,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:06:35,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:06:35,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:35,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14534.26 MB 2025-02-14 19:06:35,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14761.65 MB 2025-02-14 19:06:35,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.38 MB 2025-02-14 19:06:35,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21655.19 MB 2025-02-14 19:06:35,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21655.19 MB 2025-02-14 19:06:35,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:35,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16589.49 MB 2025-02-14 19:06:35,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:06:35,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:06:35,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.73 seconds 2025-02-14 19:06:35,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:35,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.65 MB 2025-02-14 19:06:35,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14954.08 MB 2025-02-14 19:06:35,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 19:06:35,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21655.19 MB 2025-02-14 19:06:35,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21655.19 MB 2025-02-14 19:06:35,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:35,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18931.30 MB 2025-02-14 19:06:35,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:06:35,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:06:35,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:06:35,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:35,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.01 MB 2025-02-14 19:06:35,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15638.80 MB 2025-02-14 19:06:35,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 19:06:35,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21655.19 MB 2025-02-14 19:06:35,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21655.19 MB 2025-02-14 19:06:35,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:35,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16152.63 MB 2025-02-14 19:06:36,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:06:36,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:06:36,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 19:06:36,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:36,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15638.80 MB 2025-02-14 19:06:36,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16451.52 MB 2025-02-14 19:06:36,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 19:06:36,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21655.19 MB 2025-02-14 19:06:36,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21655.19 MB 2025-02-14 19:06:36,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:36,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.28 MB 2025-02-14 19:06:36,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:06:36,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:06:36,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:06:36,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:36,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.01 MB 2025-02-14 19:06:36,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16451.52 MB 2025-02-14 19:06:36,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 19:06:36,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21655.19 MB 2025-02-14 19:06:36,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21655.19 MB 2025-02-14 19:06:36,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:36,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.28 MB 2025-02-14 19:06:36,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:06:36,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:06:36,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 19:06:36,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:36,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17007.43 MB 2025-02-14 19:06:36,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17285.47 MB 2025-02-14 19:06:36,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 19:06:36,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21655.19 MB 2025-02-14 19:06:36,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21804.09 MB 2025-02-14 19:06:36,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 19:06:36,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17553.24 MB 2025-02-14 19:06:36,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:06:36,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:06:36,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:06:36,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:36,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17435.15 MB 2025-02-14 19:06:36,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17664.61 MB 2025-02-14 19:06:36,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.47 MB 2025-02-14 19:06:36,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21804.09 MB 2025-02-14 19:06:36,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21804.09 MB 2025-02-14 19:06:36,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:36,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17664.61 MB 2025-02-14 19:06:36,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:06:36,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:06:36,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-14 19:06:36,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:36,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13487.83 MB 2025-02-14 19:06:36,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17865.56 MB 2025-02-14 19:06:36,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4377.73 MB 2025-02-14 19:06:36,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57063.51 MB 2025-02-14 19:06:36,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21804.09 MB 2025-02-14 19:06:36,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35259.42 MB 2025-02-14 19:06:36,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17865.56 MB 2025-02-14 19:06:36,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:06:36,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:06:36,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:06:36,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:36,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17865.56 MB 2025-02-14 19:06:36,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17286.90 MB 2025-02-14 19:06:36,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -578.67 MB 2025-02-14 19:06:36,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21804.09 MB 2025-02-14 19:06:36,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21804.09 MB 2025-02-14 19:06:36,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:06:36,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18970.02 MB 2025-02-14 19:06:36,409 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 19:06:36,409 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:06:36,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:06:36,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:06:36,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 19:06:36,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:06:36,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17286.90 MB 2025-02-14 19:06:36,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25721.52 MB 2025-02-14 19:06:36,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 19:06:36,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21804.09 MB 2025-02-14 19:06:36,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30188.50 MB 2025-02-14 19:06:36,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 19:06:36,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25721.52 MB 2025-02-14 19:06:36,579 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 19:06:36,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:36,580 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:06:36,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:36,581 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:06:36,586 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:06:36,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:06:36,587 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:06:36,587 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:07:27,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:27,216 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:07:27,221 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:07:27,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:27,225 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 206, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:07:27,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:27,226 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 206, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:07:30,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:07:30,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:07:30,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.18 seconds 2025-02-14 19:07:30,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:30,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.15 MB 2025-02-14 19:07:30,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15133.17 MB 2025-02-14 19:07:30,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 729.02 MB 2025-02-14 19:07:30,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38572.92 MB 2025-02-14 19:07:30,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21653.09 MB 2025-02-14 19:07:30,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16919.82 MB 2025-02-14 19:07:30,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24102.01 MB 2025-02-14 19:07:30,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:07:30,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:07:30,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:07:30,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:30,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15133.17 MB 2025-02-14 19:07:30,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15486.31 MB 2025-02-14 19:07:30,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.14 MB 2025-02-14 19:07:30,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21653.09 MB 2025-02-14 19:07:30,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21653.09 MB 2025-02-14 19:07:30,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:07:30,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18076.20 MB 2025-02-14 19:07:31,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:07:31,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:07:31,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.99 seconds 2025-02-14 19:07:31,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:31,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15486.31 MB 2025-02-14 19:07:31,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15759.70 MB 2025-02-14 19:07:31,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 273.38 MB 2025-02-14 19:07:31,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21653.09 MB 2025-02-14 19:07:31,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21653.09 MB 2025-02-14 19:07:31,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:07:31,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19740.90 MB 2025-02-14 19:07:31,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:07:31,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:07:31,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:07:31,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:31,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15759.70 MB 2025-02-14 19:07:31,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16732.57 MB 2025-02-14 19:07:31,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 972.87 MB 2025-02-14 19:07:31,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21653.09 MB 2025-02-14 19:07:31,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21653.09 MB 2025-02-14 19:07:31,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:07:31,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17462.55 MB 2025-02-14 19:07:31,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:07:31,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:07:31,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:07:31,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:31,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16732.57 MB 2025-02-14 19:07:31,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17887.16 MB 2025-02-14 19:07:31,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1154.59 MB 2025-02-14 19:07:31,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21653.09 MB 2025-02-14 19:07:31,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22139.63 MB 2025-02-14 19:07:31,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 486.54 MB 2025-02-14 19:07:31,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20742.96 MB 2025-02-14 19:07:31,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:07:31,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:07:31,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:07:31,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:31,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15759.70 MB 2025-02-14 19:07:31,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17887.16 MB 2025-02-14 19:07:31,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2127.46 MB 2025-02-14 19:07:31,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21653.09 MB 2025-02-14 19:07:31,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22139.63 MB 2025-02-14 19:07:31,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 486.54 MB 2025-02-14 19:07:31,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20742.96 MB 2025-02-14 19:07:31,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:07:31,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:07:31,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:07:31,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:31,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18676.93 MB 2025-02-14 19:07:31,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19071.94 MB 2025-02-14 19:07:31,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 395.01 MB 2025-02-14 19:07:31,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22139.63 MB 2025-02-14 19:07:31,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22353.54 MB 2025-02-14 19:07:31,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-14 19:07:31,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19437.37 MB 2025-02-14 19:07:31,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:07:31,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:07:31,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:07:31,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:31,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19284.58 MB 2025-02-14 19:07:31,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19501.73 MB 2025-02-14 19:07:31,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.15 MB 2025-02-14 19:07:31,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22353.54 MB 2025-02-14 19:07:31,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22353.54 MB 2025-02-14 19:07:31,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:07:31,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19540.84 MB 2025-02-14 19:07:31,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:07:31,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:07:31,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.41 seconds 2025-02-14 19:07:31,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:31,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13686.43 MB 2025-02-14 19:07:31,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19702.80 MB 2025-02-14 19:07:31,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6016.38 MB 2025-02-14 19:07:31,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38572.92 MB 2025-02-14 19:07:31,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22353.54 MB 2025-02-14 19:07:31,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16219.37 MB 2025-02-14 19:07:31,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19702.80 MB 2025-02-14 19:07:32,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:07:32,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:07:32,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:07:32,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:32,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.24 MB 2025-02-14 19:07:32,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17775.27 MB 2025-02-14 19:07:32,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 19:07:32,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22353.54 MB 2025-02-14 19:07:32,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22353.54 MB 2025-02-14 19:07:32,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:07:32,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18076.64 MB 2025-02-14 19:07:32,155 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:07:32,155 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:07:32,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:07:32,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:07:32,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:07:32,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:07:32,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17775.27 MB 2025-02-14 19:07:32,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26214.30 MB 2025-02-14 19:07:32,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:07:32,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22353.54 MB 2025-02-14 19:07:32,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30744.25 MB 2025-02-14 19:07:32,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:07:32,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26214.30 MB 2025-02-14 19:07:32,320 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:07:32,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:32,322 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:07:32,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:32,323 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:07:32,328 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:07:32,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:32,329 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:07:32,329 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:07:43,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:43,408 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:07:43,413 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:07:43,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:43,417 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1285, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:07:43,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:07:43,418 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1285, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:08:03,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:08:03,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:08:03,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.88 seconds 2025-02-14 19:08:03,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:03,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21922.79 MB 2025-02-14 19:08:03,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.34 MB 2025-02-14 19:08:03,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4547.54 MB 2025-02-14 19:08:03,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43329.26 MB 2025-02-14 19:08:03,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34590.43 MB 2025-02-14 19:08:03,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8738.83 MB 2025-02-14 19:08:03,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.03 MB 2025-02-14 19:08:03,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:08:03,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:08:03,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 19:08:03,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:03,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26470.34 MB 2025-02-14 19:08:03,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.17 MB 2025-02-14 19:08:03,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4012.17 MB 2025-02-14 19:08:03,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34590.43 MB 2025-02-14 19:08:03,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44828.72 MB 2025-02-14 19:08:03,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10238.30 MB 2025-02-14 19:08:03,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40019.54 MB 2025-02-14 19:08:05,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:08:05,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:08:05,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 19:08:05,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.17 MB 2025-02-14 19:08:05,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22989.01 MB 2025-02-14 19:08:05,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:08:05,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44828.72 MB 2025-02-14 19:08:05,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30041.70 MB 2025-02-14 19:08:05,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14787.02 MB 2025-02-14 19:08:05,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26967.55 MB 2025-02-14 19:08:05,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:08:05,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:08:05,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:08:05,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 19:08:05,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.54 MB 2025-02-14 19:08:05,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:08:05,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30041.70 MB 2025-02-14 19:08:05,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30041.70 MB 2025-02-14 19:08:05,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:05,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26295.97 MB 2025-02-14 19:08:05,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:08:05,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:08:05,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:08:05,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.54 MB 2025-02-14 19:08:05,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 19:08:05,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:08:05,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30041.70 MB 2025-02-14 19:08:05,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34760.29 MB 2025-02-14 19:08:05,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 19:08:05,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 19:08:05,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:08:05,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:08:05,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:08:05,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 19:08:05,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 19:08:05,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:08:05,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30041.70 MB 2025-02-14 19:08:05,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34760.29 MB 2025-02-14 19:08:05,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 19:08:05,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 19:08:05,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:08:05,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:08:05,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:08:05,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28653.94 MB 2025-02-14 19:08:05,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29420.94 MB 2025-02-14 19:08:05,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:08:05,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34760.29 MB 2025-02-14 19:08:05,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35177.63 MB 2025-02-14 19:08:05,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:08:05,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30128.73 MB 2025-02-14 19:08:05,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:08:05,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:08:05,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:08:05,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29833.83 MB 2025-02-14 19:08:05,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30061.43 MB 2025-02-14 19:08:05,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.60 MB 2025-02-14 19:08:05,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35177.63 MB 2025-02-14 19:08:05,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35177.63 MB 2025-02-14 19:08:05,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:05,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30299.32 MB 2025-02-14 19:08:05,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:08:05,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:08:05,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.30 seconds 2025-02-14 19:08:05,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17445.75 MB 2025-02-14 19:08:05,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30261.66 MB 2025-02-14 19:08:05,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12815.91 MB 2025-02-14 19:08:05,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43329.26 MB 2025-02-14 19:08:05,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35177.63 MB 2025-02-14 19:08:05,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8151.63 MB 2025-02-14 19:08:05,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30299.32 MB 2025-02-14 19:08:05,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:08:05,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:08:05,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:08:05,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:05,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30261.66 MB 2025-02-14 19:08:05,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22437.43 MB 2025-02-14 19:08:05,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7824.23 MB 2025-02-14 19:08:05,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35177.63 MB 2025-02-14 19:08:05,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35177.63 MB 2025-02-14 19:08:05,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:05,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30261.66 MB 2025-02-14 19:08:06,007 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 19:08:06,007 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:08:06,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:08:06,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:08:06,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:08:06,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:06,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22437.43 MB 2025-02-14 19:08:06,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30842.51 MB 2025-02-14 19:08:06,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-14 19:08:06,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35177.63 MB 2025-02-14 19:08:06,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43532.68 MB 2025-02-14 19:08:06,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 19:08:06,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30842.51 MB 2025-02-14 19:08:06,172 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 19:08:06,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:06,173 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:08:06,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:06,174 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:08:06,179 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:08:06,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:06,180 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:08:06,180 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:08:18,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:18,849 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:08:18,854 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:08:18,859 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:18,859 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:08:18,860 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:18,860 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:08:22,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:08:22,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:08:22,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.20 seconds 2025-02-14 19:08:22,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:22,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14390.21 MB 2025-02-14 19:08:22,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15112.16 MB 2025-02-14 19:08:22,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-14 19:08:22,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51887.73 MB 2025-02-14 19:08:22,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20000.54 MB 2025-02-14 19:08:22,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31887.20 MB 2025-02-14 19:08:22,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.07 MB 2025-02-14 19:08:22,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:08:22,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:08:22,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:08:22,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:22,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15112.16 MB 2025-02-14 19:08:22,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15384.68 MB 2025-02-14 19:08:22,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.53 MB 2025-02-14 19:08:22,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20000.54 MB 2025-02-14 19:08:22,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20000.54 MB 2025-02-14 19:08:22,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:22,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17865.58 MB 2025-02-14 19:08:23,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:08:23,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:08:23,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-14 19:08:23,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15384.68 MB 2025-02-14 19:08:23,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15640.81 MB 2025-02-14 19:08:23,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 19:08:23,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20000.54 MB 2025-02-14 19:08:23,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 19:08:23,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 19:08:23,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19640.31 MB 2025-02-14 19:08:23,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:08:23,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:08:23,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:08:23,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-14 19:08:23,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16552.23 MB 2025-02-14 19:08:23,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 19:08:23,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 19:08:23,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 19:08:23,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:23,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17236.14 MB 2025-02-14 19:08:23,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:08:23,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:08:23,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 19:08:23,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16552.23 MB 2025-02-14 19:08:23,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-14 19:08:23,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-14 19:08:23,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 19:08:23,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21814.58 MB 2025-02-14 19:08:23,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-14 19:08:23,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20310.87 MB 2025-02-14 19:08:23,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:08:23,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:08:23,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 19:08:23,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15640.75 MB 2025-02-14 19:08:23,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.96 MB 2025-02-14 19:08:23,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-14 19:08:23,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 19:08:23,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21814.58 MB 2025-02-14 19:08:23,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-14 19:08:23,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20310.87 MB 2025-02-14 19:08:23,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:08:23,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:08:23,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 19:08:23,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18373.89 MB 2025-02-14 19:08:23,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18745.80 MB 2025-02-14 19:08:23,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.91 MB 2025-02-14 19:08:23,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21814.58 MB 2025-02-14 19:08:23,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22015.90 MB 2025-02-14 19:08:23,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-14 19:08:23,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19090.09 MB 2025-02-14 19:08:23,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:08:23,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:08:23,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:08:23,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18945.03 MB 2025-02-14 19:08:23,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19172.31 MB 2025-02-14 19:08:23,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.28 MB 2025-02-14 19:08:23,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22015.90 MB 2025-02-14 19:08:23,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22015.90 MB 2025-02-14 19:08:23,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:23,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19202.95 MB 2025-02-14 19:08:23,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:08:23,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:08:23,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.48 seconds 2025-02-14 19:08:23,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13679.46 MB 2025-02-14 19:08:23,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19373.38 MB 2025-02-14 19:08:23,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5693.92 MB 2025-02-14 19:08:23,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51887.73 MB 2025-02-14 19:08:23,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22015.90 MB 2025-02-14 19:08:23,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29871.83 MB 2025-02-14 19:08:23,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19373.38 MB 2025-02-14 19:08:23,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:08:23,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:08:23,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 19:08:23,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19373.38 MB 2025-02-14 19:08:23,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17708.79 MB 2025-02-14 19:08:23,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1664.59 MB 2025-02-14 19:08:23,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22015.90 MB 2025-02-14 19:08:23,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22015.90 MB 2025-02-14 19:08:23,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:23,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19373.38 MB 2025-02-14 19:08:23,652 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:08:23,652 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:08:23,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:08:23,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:08:23,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:08:23,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:23,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17708.79 MB 2025-02-14 19:08:23,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26147.81 MB 2025-02-14 19:08:23,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:08:23,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22015.90 MB 2025-02-14 19:08:23,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30406.61 MB 2025-02-14 19:08:23,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:08:23,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26147.81 MB 2025-02-14 19:08:23,909 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:08:23,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:23,911 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:08:23,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:23,913 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:08:23,921 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:08:23,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:23,923 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:08:23,923 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:08:41,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:41,149 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:08:41,154 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:08:41,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:41,158 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:08:41,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:41,159 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:08:45,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:08:45,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:08:45,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.52 seconds 2025-02-14 19:08:45,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:45,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14989.47 MB 2025-02-14 19:08:45,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16015.77 MB 2025-02-14 19:08:45,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1026.29 MB 2025-02-14 19:08:45,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42991.62 MB 2025-02-14 19:08:45,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20694.70 MB 2025-02-14 19:08:45,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22296.92 MB 2025-02-14 19:08:45,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24913.83 MB 2025-02-14 19:08:45,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:08:45,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:08:45,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:08:45,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:45,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16015.77 MB 2025-02-14 19:08:45,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16506.90 MB 2025-02-14 19:08:45,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 491.13 MB 2025-02-14 19:08:45,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20694.70 MB 2025-02-14 19:08:45,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22726.84 MB 2025-02-14 19:08:45,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2032.14 MB 2025-02-14 19:08:45,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20083.13 MB 2025-02-14 19:08:47,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:08:47,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:08:47,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.38 seconds 2025-02-14 19:08:47,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16506.90 MB 2025-02-14 19:08:47,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16890.43 MB 2025-02-14 19:08:47,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 383.53 MB 2025-02-14 19:08:47,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22726.84 MB 2025-02-14 19:08:47,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21088.96 MB 2025-02-14 19:08:47,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-14 19:08:47,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20847.46 MB 2025-02-14 19:08:47,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:08:47,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:08:47,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:08:47,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16890.43 MB 2025-02-14 19:08:47,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18255.61 MB 2025-02-14 19:08:47,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1365.18 MB 2025-02-14 19:08:47,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21088.96 MB 2025-02-14 19:08:47,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21088.96 MB 2025-02-14 19:08:47,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:47,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19279.70 MB 2025-02-14 19:08:47,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:08:47,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:08:47,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 19:08:47,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18255.61 MB 2025-02-14 19:08:47,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19875.37 MB 2025-02-14 19:08:47,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1619.76 MB 2025-02-14 19:08:47,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21088.96 MB 2025-02-14 19:08:47,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25190.99 MB 2025-02-14 19:08:47,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4102.03 MB 2025-02-14 19:08:47,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23881.09 MB 2025-02-14 19:08:47,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:08:47,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:08:47,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:08:47,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16890.43 MB 2025-02-14 19:08:47,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19875.37 MB 2025-02-14 19:08:47,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2984.94 MB 2025-02-14 19:08:47,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21088.96 MB 2025-02-14 19:08:47,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25190.99 MB 2025-02-14 19:08:47,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4102.03 MB 2025-02-14 19:08:47,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23881.09 MB 2025-02-14 19:08:47,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:08:47,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:08:47,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:08:47,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20983.35 MB 2025-02-14 19:08:47,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21537.51 MB 2025-02-14 19:08:47,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 554.16 MB 2025-02-14 19:08:47,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25190.99 MB 2025-02-14 19:08:47,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25490.88 MB 2025-02-14 19:08:47,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 299.89 MB 2025-02-14 19:08:47,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22048.89 MB 2025-02-14 19:08:47,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:08:47,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:08:47,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:08:47,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21835.83 MB 2025-02-14 19:08:47,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22066.54 MB 2025-02-14 19:08:47,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.71 MB 2025-02-14 19:08:47,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25490.88 MB 2025-02-14 19:08:47,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25490.88 MB 2025-02-14 19:08:47,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:08:47,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22168.24 MB 2025-02-14 19:08:47,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:08:47,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:08:47,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.24 seconds 2025-02-14 19:08:47,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-14 19:08:47,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22267.61 MB 2025-02-14 19:08:47,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8288.52 MB 2025-02-14 19:08:47,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42991.62 MB 2025-02-14 19:08:47,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25490.88 MB 2025-02-14 19:08:47,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17500.73 MB 2025-02-14 19:08:47,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22267.61 MB 2025-02-14 19:08:47,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:08:47,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:08:47,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:08:47,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22267.61 MB 2025-02-14 19:08:47,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25281.64 MB 2025-02-14 19:08:47,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 19:08:47,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25490.88 MB 2025-02-14 19:08:47,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26833.06 MB 2025-02-14 19:08:47,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1342.18 MB 2025-02-14 19:08:47,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25583.27 MB 2025-02-14 19:08:47,689 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:08:47,689 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 19:08:47,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:08:47,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:08:47,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:08:47,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:08:47,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18459.38 MB 2025-02-14 19:08:47,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26898.40 MB 2025-02-14 19:08:47,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:08:47,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26833.06 MB 2025-02-14 19:08:47,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35223.76 MB 2025-02-14 19:08:47,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:08:47,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26898.40 MB 2025-02-14 19:08:47,861 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:08:47,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:47,863 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:08:47,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:47,864 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:08:47,868 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:08:47,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:08:47,869 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:08:47,870 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 19:10:30,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:30,104 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:10:30,109 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:10:30,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:30,113 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 332, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:10:30,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:30,114 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 332, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:10:35,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:10:35,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:10:35,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.09 seconds 2025-02-14 19:10:35,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:35,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15282.14 MB 2025-02-14 19:10:35,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16457.07 MB 2025-02-14 19:10:35,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1174.93 MB 2025-02-14 19:10:35,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47808.77 MB 2025-02-14 19:10:35,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24125.64 MB 2025-02-14 19:10:35,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23683.14 MB 2025-02-14 19:10:35,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25432.98 MB 2025-02-14 19:10:35,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:10:35,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:10:35,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:10:35,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:35,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16457.07 MB 2025-02-14 19:10:35,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17026.25 MB 2025-02-14 19:10:35,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.18 MB 2025-02-14 19:10:35,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24125.64 MB 2025-02-14 19:10:35,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24125.64 MB 2025-02-14 19:10:35,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:10:35,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21162.82 MB 2025-02-14 19:10:36,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:10:36,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:10:36,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.58 seconds 2025-02-14 19:10:36,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:36,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17026.25 MB 2025-02-14 19:10:36,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17466.85 MB 2025-02-14 19:10:36,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.60 MB 2025-02-14 19:10:36,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24125.64 MB 2025-02-14 19:10:36,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23441.97 MB 2025-02-14 19:10:36,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -683.67 MB 2025-02-14 19:10:36,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21450.70 MB 2025-02-14 19:10:36,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:10:36,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:10:36,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:10:36,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:36,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17466.85 MB 2025-02-14 19:10:36,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19035.52 MB 2025-02-14 19:10:36,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1568.67 MB 2025-02-14 19:10:36,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23441.97 MB 2025-02-14 19:10:36,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23441.97 MB 2025-02-14 19:10:36,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:10:36,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20211.99 MB 2025-02-14 19:10:36,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:10:36,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:10:36,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 19:10:36,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:36,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19035.52 MB 2025-02-14 19:10:36,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20896.27 MB 2025-02-14 19:10:36,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1860.75 MB 2025-02-14 19:10:36,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23441.97 MB 2025-02-14 19:10:36,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27363.64 MB 2025-02-14 19:10:36,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3921.67 MB 2025-02-14 19:10:36,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25503.26 MB 2025-02-14 19:10:36,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:10:36,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:10:36,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 19:10:36,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:36,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17466.85 MB 2025-02-14 19:10:36,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20896.27 MB 2025-02-14 19:10:36,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3429.42 MB 2025-02-14 19:10:36,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23441.97 MB 2025-02-14 19:10:36,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27363.64 MB 2025-02-14 19:10:36,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3921.67 MB 2025-02-14 19:10:36,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25503.26 MB 2025-02-14 19:10:37,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:10:37,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:10:37,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 19:10:37,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:37,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22169.11 MB 2025-02-14 19:10:37,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22806.25 MB 2025-02-14 19:10:37,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.14 MB 2025-02-14 19:10:37,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27363.64 MB 2025-02-14 19:10:37,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 19:10:37,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 343.93 MB 2025-02-14 19:10:37,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23393.71 MB 2025-02-14 19:10:37,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:10:37,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:10:37,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:10:37,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:37,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23148.95 MB 2025-02-14 19:10:37,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23363.48 MB 2025-02-14 19:10:37,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.53 MB 2025-02-14 19:10:37,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 19:10:37,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 19:10:37,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:10:37,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23508.61 MB 2025-02-14 19:10:37,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:10:37,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:10:37,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.04 seconds 2025-02-14 19:10:37,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:37,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14125.42 MB 2025-02-14 19:10:37,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23564.55 MB 2025-02-14 19:10:37,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9439.13 MB 2025-02-14 19:10:37,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47808.77 MB 2025-02-14 19:10:37,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 19:10:37,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20101.20 MB 2025-02-14 19:10:37,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23564.55 MB 2025-02-14 19:10:37,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:10:37,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:10:37,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:10:37,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:37,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23564.55 MB 2025-02-14 19:10:37,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26578.58 MB 2025-02-14 19:10:37,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 19:10:37,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 19:10:37,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27976.01 MB 2025-02-14 19:10:37,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 19:10:37,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26880.21 MB 2025-02-14 19:10:37,450 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:10:37,450 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:10:37,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:10:37,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:10:37,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:10:37,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:10:37,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18809.42 MB 2025-02-14 19:10:37,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27248.45 MB 2025-02-14 19:10:37,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:10:37,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27976.01 MB 2025-02-14 19:10:37,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36366.71 MB 2025-02-14 19:10:37,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:10:37,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27248.45 MB 2025-02-14 19:10:37,616 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:10:37,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:37,617 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:10:37,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:37,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:10:37,623 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:10:37,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:37,624 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:10:37,624 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:10:58,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:58,254 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:10:58,259 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:10:58,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:58,262 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2332, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:10:58,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:10:58,263 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2332, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:11:34,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:11:34,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:11:34,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.08 seconds 2025-02-14 19:11:34,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:34,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29218.46 MB 2025-02-14 19:11:34,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37471.27 MB 2025-02-14 19:11:34,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8252.82 MB 2025-02-14 19:11:34,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48951.72 MB 2025-02-14 19:11:34,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38952.50 MB 2025-02-14 19:11:34,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9999.22 MB 2025-02-14 19:11:34,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46391.38 MB 2025-02-14 19:11:34,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:11:34,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:11:34,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 19:11:34,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:34,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37471.27 MB 2025-02-14 19:11:34,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27902.24 MB 2025-02-14 19:11:34,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9569.03 MB 2025-02-14 19:11:34,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38952.50 MB 2025-02-14 19:11:34,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 73264.01 MB 2025-02-14 19:11:34,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 34311.50 MB 2025-02-14 19:11:34,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60283.75 MB 2025-02-14 19:11:36,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:11:36,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:11:36,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 19:11:36,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:36,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27902.24 MB 2025-02-14 19:11:36,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28433.08 MB 2025-02-14 19:11:36,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:11:36,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73264.01 MB 2025-02-14 19:11:36,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33032.24 MB 2025-02-14 19:11:36,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40231.76 MB 2025-02-14 19:11:36,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32411.63 MB 2025-02-14 19:11:36,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:11:36,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:11:36,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:11:36,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:36,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28433.08 MB 2025-02-14 19:11:36,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.62 MB 2025-02-14 19:11:36,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:11:36,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33032.24 MB 2025-02-14 19:11:36,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-14 19:11:36,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 19:11:36,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31740.04 MB 2025-02-14 19:11:36,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:11:36,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:11:36,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:11:36,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:36,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30322.62 MB 2025-02-14 19:11:36,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32564.47 MB 2025-02-14 19:11:36,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:11:36,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-14 19:11:36,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40110.13 MB 2025-02-14 19:11:36,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:11:36,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38108.75 MB 2025-02-14 19:11:36,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:11:36,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:11:36,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:11:36,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:36,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28433.08 MB 2025-02-14 19:11:36,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32564.47 MB 2025-02-14 19:11:36,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:11:36,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33032.24 MB 2025-02-14 19:11:36,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40110.13 MB 2025-02-14 19:11:36,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 19:11:36,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38108.75 MB 2025-02-14 19:11:36,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:11:36,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:11:36,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:11:36,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:36,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34098.01 MB 2025-02-14 19:11:36,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34865.02 MB 2025-02-14 19:11:36,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:11:36,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40110.13 MB 2025-02-14 19:11:36,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-14 19:11:36,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:11:36,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35572.80 MB 2025-02-14 19:11:36,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:11:36,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:11:36,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:11:36,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:36,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35277.90 MB 2025-02-14 19:11:36,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35507.50 MB 2025-02-14 19:11:36,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.60 MB 2025-02-14 19:11:36,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-14 19:11:36,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-14 19:11:36,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:11:36,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35719.07 MB 2025-02-14 19:11:36,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:11:36,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:11:36,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.61 seconds 2025-02-14 19:11:36,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:36,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21093.58 MB 2025-02-14 19:11:36,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35708.58 MB 2025-02-14 19:11:36,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14615.00 MB 2025-02-14 19:11:36,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48951.72 MB 2025-02-14 19:11:36,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-14 19:11:36,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8426.36 MB 2025-02-14 19:11:36,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35719.07 MB 2025-02-14 19:11:37,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:11:37,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:11:37,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:11:37,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:37,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35708.58 MB 2025-02-14 19:11:37,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26097.97 MB 2025-02-14 19:11:37,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9610.61 MB 2025-02-14 19:11:37,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-14 19:11:37,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-14 19:11:37,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:11:37,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38220.24 MB 2025-02-14 19:11:37,164 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:11:37,165 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:11:37,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:11:37,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:11:37,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:11:37,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:11:37,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26097.97 MB 2025-02-14 19:11:37,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34536.99 MB 2025-02-14 19:11:37,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:11:37,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40525.37 MB 2025-02-14 19:11:37,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48916.07 MB 2025-02-14 19:11:37,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:11:37,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34536.99 MB 2025-02-14 19:11:37,327 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:11:37,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:11:37,329 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:11:37,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:11:37,329 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:11:37,334 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:11:37,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:11:37,335 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:11:37,335 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:12:15,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:15,794 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:12:15,799 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:12:15,802 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:15,802 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:12:15,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:15,803 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:12:21,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:12:21,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:12:21,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.85 seconds 2025-02-14 19:12:21,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:21,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-14 19:12:21,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16950.90 MB 2025-02-14 19:12:21,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1341.26 MB 2025-02-14 19:12:21,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61501.08 MB 2025-02-14 19:12:21,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24698.16 MB 2025-02-14 19:12:21,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36802.92 MB 2025-02-14 19:12:21,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25760.49 MB 2025-02-14 19:12:21,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:12:21,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:12:21,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:12:21,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:21,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16950.90 MB 2025-02-14 19:12:21,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17389.98 MB 2025-02-14 19:12:21,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.08 MB 2025-02-14 19:12:21,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24698.16 MB 2025-02-14 19:12:21,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24698.16 MB 2025-02-14 19:12:21,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:12:21,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21891.90 MB 2025-02-14 19:12:23,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:12:23,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:12:23,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.66 seconds 2025-02-14 19:12:23,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17389.98 MB 2025-02-14 19:12:23,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17853.14 MB 2025-02-14 19:12:23,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 463.16 MB 2025-02-14 19:12:23,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24698.16 MB 2025-02-14 19:12:23,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23989.32 MB 2025-02-14 19:12:23,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -708.84 MB 2025-02-14 19:12:23,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21814.43 MB 2025-02-14 19:12:23,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:12:23,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:12:23,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:12:23,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17853.14 MB 2025-02-14 19:12:23,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19502.02 MB 2025-02-14 19:12:23,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1648.89 MB 2025-02-14 19:12:23,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23989.32 MB 2025-02-14 19:12:23,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23989.32 MB 2025-02-14 19:12:23,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:12:23,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20738.73 MB 2025-02-14 19:12:23,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:12:23,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:12:23,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 19:12:23,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19502.02 MB 2025-02-14 19:12:23,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21458.05 MB 2025-02-14 19:12:23,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1956.03 MB 2025-02-14 19:12:23,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23989.32 MB 2025-02-14 19:12:23,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28525.46 MB 2025-02-14 19:12:23,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4536.14 MB 2025-02-14 19:12:23,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26298.58 MB 2025-02-14 19:12:23,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:12:23,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:12:23,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:12:23,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17853.14 MB 2025-02-14 19:12:23,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21458.05 MB 2025-02-14 19:12:23,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3604.91 MB 2025-02-14 19:12:23,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23989.32 MB 2025-02-14 19:12:23,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28525.46 MB 2025-02-14 19:12:23,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4536.14 MB 2025-02-14 19:12:23,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26298.58 MB 2025-02-14 19:12:23,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:12:23,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:12:23,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 19:12:23,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.07 MB 2025-02-14 19:12:23,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23465.28 MB 2025-02-14 19:12:23,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 669.21 MB 2025-02-14 19:12:23,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28525.46 MB 2025-02-14 19:12:23,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28884.07 MB 2025-02-14 19:12:23,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 358.61 MB 2025-02-14 19:12:23,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24082.82 MB 2025-02-14 19:12:23,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:12:23,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:12:23,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:12:23,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23825.52 MB 2025-02-14 19:12:23,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24043.88 MB 2025-02-14 19:12:23,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.36 MB 2025-02-14 19:12:23,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28884.07 MB 2025-02-14 19:12:23,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28884.07 MB 2025-02-14 19:12:23,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:12:23,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24217.76 MB 2025-02-14 19:12:23,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:12:23,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:12:23,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.90 seconds 2025-02-14 19:12:23,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-14 19:12:23,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24244.96 MB 2025-02-14 19:12:23,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9955.78 MB 2025-02-14 19:12:23,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61501.08 MB 2025-02-14 19:12:23,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28884.07 MB 2025-02-14 19:12:23,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32617.01 MB 2025-02-14 19:12:23,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24244.96 MB 2025-02-14 19:12:23,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:12:23,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:12:23,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:12:23,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24244.96 MB 2025-02-14 19:12:23,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27258.99 MB 2025-02-14 19:12:23,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 19:12:23,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28884.07 MB 2025-02-14 19:12:23,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28884.07 MB 2025-02-14 19:12:23,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:12:23,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27560.36 MB 2025-02-14 19:12:23,987 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:12:23,987 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:12:23,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:12:23,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:12:23,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:12:23,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:23,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19052.88 MB 2025-02-14 19:12:23,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27491.90 MB 2025-02-14 19:12:23,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:12:23,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28884.07 MB 2025-02-14 19:12:23,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37274.78 MB 2025-02-14 19:12:23,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:12:23,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27491.90 MB 2025-02-14 19:12:24,156 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:12:24,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:24,157 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:12:24,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:24,158 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:12:24,163 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:12:24,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:24,164 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:12:24,164 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:12:35,298 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:35,298 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:12:35,303 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:12:35,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:35,307 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1047, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:12:35,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:35,308 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1047, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:12:51,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:12:51,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:12:51,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.32 seconds 2025-02-14 19:12:51,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:51,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20264.37 MB 2025-02-14 19:12:51,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23970.04 MB 2025-02-14 19:12:51,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3705.67 MB 2025-02-14 19:12:51,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49859.79 MB 2025-02-14 19:12:51,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29288.82 MB 2025-02-14 19:12:51,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20570.96 MB 2025-02-14 19:12:51,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32907.93 MB 2025-02-14 19:12:51,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:12:51,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:12:51,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 19:12:51,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:51,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23970.04 MB 2025-02-14 19:12:51,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21221.93 MB 2025-02-14 19:12:51,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2748.11 MB 2025-02-14 19:12:51,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29288.82 MB 2025-02-14 19:12:51,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35930.51 MB 2025-02-14 19:12:51,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6641.68 MB 2025-02-14 19:12:51,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32365.02 MB 2025-02-14 19:12:53,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:12:53,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:12:53,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 19:12:53,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:53,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21221.93 MB 2025-02-14 19:12:53,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21752.77 MB 2025-02-14 19:12:53,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:12:53,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35930.51 MB 2025-02-14 19:12:53,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 19:12:53,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8222.93 MB 2025-02-14 19:12:53,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25731.31 MB 2025-02-14 19:12:53,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:12:53,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:12:53,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:12:53,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:53,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21752.77 MB 2025-02-14 19:12:53,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23642.30 MB 2025-02-14 19:12:53,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:12:53,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 19:12:53,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27707.57 MB 2025-02-14 19:12:53,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:12:53,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25059.73 MB 2025-02-14 19:12:53,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:12:53,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:12:53,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:12:53,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:53,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23642.30 MB 2025-02-14 19:12:53,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25884.16 MB 2025-02-14 19:12:53,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:12:53,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 19:12:53,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33369.88 MB 2025-02-14 19:12:53,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:12:53,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31428.44 MB 2025-02-14 19:12:53,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:12:53,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:12:53,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:12:53,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:53,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21752.77 MB 2025-02-14 19:12:53,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25884.16 MB 2025-02-14 19:12:53,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:12:53,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27707.57 MB 2025-02-14 19:12:53,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33369.88 MB 2025-02-14 19:12:53,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:12:53,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31428.44 MB 2025-02-14 19:12:53,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:12:53,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:12:53,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:12:53,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:53,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27417.70 MB 2025-02-14 19:12:53,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28184.70 MB 2025-02-14 19:12:53,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:12:53,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33369.88 MB 2025-02-14 19:12:53,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 19:12:53,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:12:53,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.49 MB 2025-02-14 19:12:54,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:12:54,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:12:54,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:12:54,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:54,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28597.59 MB 2025-02-14 19:12:54,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28829.29 MB 2025-02-14 19:12:54,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.70 MB 2025-02-14 19:12:54,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33787.22 MB 2025-02-14 19:12:54,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 19:12:54,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:12:54,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29016.69 MB 2025-02-14 19:12:54,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:12:54,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:12:54,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.71 seconds 2025-02-14 19:12:54,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:54,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16616.54 MB 2025-02-14 19:12:54,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29030.36 MB 2025-02-14 19:12:54,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12413.83 MB 2025-02-14 19:12:54,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49859.79 MB 2025-02-14 19:12:54,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 19:12:54,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16072.57 MB 2025-02-14 19:12:54,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29030.36 MB 2025-02-14 19:12:54,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:12:54,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:12:54,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:12:54,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:54,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29030.36 MB 2025-02-14 19:12:54,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21620.93 MB 2025-02-14 19:12:54,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7409.44 MB 2025-02-14 19:12:54,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33787.22 MB 2025-02-14 19:12:54,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33787.22 MB 2025-02-14 19:12:54,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:12:54,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31542.03 MB 2025-02-14 19:12:54,309 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:12:54,309 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:12:54,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:12:54,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:12:54,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:12:54,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:12:54,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21620.93 MB 2025-02-14 19:12:54,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30059.95 MB 2025-02-14 19:12:54,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:12:54,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33787.22 MB 2025-02-14 19:12:54,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42177.92 MB 2025-02-14 19:12:54,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:12:54,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30059.95 MB 2025-02-14 19:12:54,472 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:12:54,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:54,474 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:12:54,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:54,475 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:12:54,479 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:12:54,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:12:54,480 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:12:54,480 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:14:57,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:14:57,083 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:14:57,088 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:14:57,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:14:57,092 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 223, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:14:57,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:14:57,093 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 223, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:15:00,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:15:00,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:15:00,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.42 seconds 2025-02-14 19:15:00,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:00,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14522.61 MB 2025-02-14 19:15:00,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.79 MB 2025-02-14 19:15:00,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 789.18 MB 2025-02-14 19:15:00,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54762.93 MB 2025-02-14 19:15:00,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-14 19:15:00,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34993.08 MB 2025-02-14 19:15:00,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24220.47 MB 2025-02-14 19:15:00,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:15:00,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:15:00,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:15:00,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:00,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.79 MB 2025-02-14 19:15:00,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15567.67 MB 2025-02-14 19:15:00,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.88 MB 2025-02-14 19:15:00,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-14 19:15:00,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19769.85 MB 2025-02-14 19:15:00,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:15:00,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18244.31 MB 2025-02-14 19:15:01,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:15:01,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:15:01,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.98 seconds 2025-02-14 19:15:01,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15567.67 MB 2025-02-14 19:15:01,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15839.73 MB 2025-02-14 19:15:01,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.06 MB 2025-02-14 19:15:01,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19769.85 MB 2025-02-14 19:15:01,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20172.51 MB 2025-02-14 19:15:01,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 19:15:01,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19823.29 MB 2025-02-14 19:15:01,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:15:01,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:15:01,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:15:01,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15839.73 MB 2025-02-14 19:15:01,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16807.88 MB 2025-02-14 19:15:01,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 968.15 MB 2025-02-14 19:15:01,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20172.51 MB 2025-02-14 19:15:01,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20172.51 MB 2025-02-14 19:15:01,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:15:01,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17534.31 MB 2025-02-14 19:15:01,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:15:01,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:15:01,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:15:01,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16807.88 MB 2025-02-14 19:15:01,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17956.86 MB 2025-02-14 19:15:01,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1148.98 MB 2025-02-14 19:15:01,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20172.51 MB 2025-02-14 19:15:01,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22594.72 MB 2025-02-14 19:15:01,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2422.21 MB 2025-02-14 19:15:01,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20801.42 MB 2025-02-14 19:15:01,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:15:01,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:15:01,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:15:01,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15839.73 MB 2025-02-14 19:15:01,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17956.86 MB 2025-02-14 19:15:01,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2117.13 MB 2025-02-14 19:15:01,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20172.51 MB 2025-02-14 19:15:01,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22594.72 MB 2025-02-14 19:15:01,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2422.21 MB 2025-02-14 19:15:01,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20801.42 MB 2025-02-14 19:15:01,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:15:01,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:15:01,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:15:01,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.80 MB 2025-02-14 19:15:01,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19135.89 MB 2025-02-14 19:15:01,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 393.09 MB 2025-02-14 19:15:01,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22594.72 MB 2025-02-14 19:15:01,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22802.33 MB 2025-02-14 19:15:01,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-14 19:15:01,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19500.75 MB 2025-02-14 19:15:01,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:15:01,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:15:01,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:15:01,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19347.50 MB 2025-02-14 19:15:01,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19564.38 MB 2025-02-14 19:15:01,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.88 MB 2025-02-14 19:15:01,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22802.33 MB 2025-02-14 19:15:01,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22802.33 MB 2025-02-14 19:15:01,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:15:01,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19600.29 MB 2025-02-14 19:15:01,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:15:01,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:15:01,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.63 seconds 2025-02-14 19:15:01,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13745.66 MB 2025-02-14 19:15:01,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19764.88 MB 2025-02-14 19:15:01,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6019.23 MB 2025-02-14 19:15:01,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54762.93 MB 2025-02-14 19:15:01,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 19:15:01,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31958.50 MB 2025-02-14 19:15:01,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19764.88 MB 2025-02-14 19:15:01,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:15:01,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:15:01,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 19:15:01,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:01,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14815.47 MB 2025-02-14 19:15:01,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17821.02 MB 2025-02-14 19:15:01,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-14 19:15:01,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 19:15:01,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 19:15:01,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:15:01,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18121.54 MB 2025-02-14 19:15:02,013 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 19:15:02,013 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:15:02,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:15:02,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:15:02,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:15:02,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:15:02,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17821.02 MB 2025-02-14 19:15:02,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26235.97 MB 2025-02-14 19:15:02,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 19:15:02,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 19:15:02,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31172.07 MB 2025-02-14 19:15:02,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 19:15:02,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26235.97 MB 2025-02-14 19:15:02,188 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 19:15:02,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:15:02,189 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:15:02,190 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:15:02,191 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:15:02,195 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:15:02,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:15:02,196 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:15:02,196 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:16:10,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:16:10,156 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:16:10,161 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:16:10,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:16:10,166 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2715, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:16:10,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:16:10,167 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2715, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:16:51,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:16:51,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:16:51,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.51 seconds 2025-02-14 19:16:51,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:51,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31889.21 MB 2025-02-14 19:16:51,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41498.36 MB 2025-02-14 19:16:51,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9609.15 MB 2025-02-14 19:16:51,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58460.21 MB 2025-02-14 19:16:51,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46644.85 MB 2025-02-14 19:16:51,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11815.35 MB 2025-02-14 19:16:51,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51106.59 MB 2025-02-14 19:16:51,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:16:51,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:16:51,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:16:51,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:51,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41498.36 MB 2025-02-14 19:16:51,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29893.78 MB 2025-02-14 19:16:51,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11604.58 MB 2025-02-14 19:16:51,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46644.85 MB 2025-02-14 19:16:51,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77416.37 MB 2025-02-14 19:16:51,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30771.51 MB 2025-02-14 19:16:51,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67098.66 MB 2025-02-14 19:16:53,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:16:53,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:16:53,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 19:16:53,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:53,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29893.78 MB 2025-02-14 19:16:53,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30424.62 MB 2025-02-14 19:16:53,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:16:53,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77416.37 MB 2025-02-14 19:16:53,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33646.71 MB 2025-02-14 19:16:53,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -43769.66 MB 2025-02-14 19:16:53,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34404.21 MB 2025-02-14 19:16:53,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:16:53,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:16:53,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:16:53,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:53,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30424.62 MB 2025-02-14 19:16:53,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32314.16 MB 2025-02-14 19:16:53,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:16:53,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33646.71 MB 2025-02-14 19:16:53,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35534.14 MB 2025-02-14 19:16:53,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:16:53,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33731.59 MB 2025-02-14 19:16:54,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:16:54,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:16:54,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 19:16:54,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:54,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32314.16 MB 2025-02-14 19:16:54,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34556.01 MB 2025-02-14 19:16:54,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:16:54,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35534.14 MB 2025-02-14 19:16:54,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41668.31 MB 2025-02-14 19:16:54,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:16:54,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40100.29 MB 2025-02-14 19:16:54,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:16:54,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:16:54,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 19:16:54,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:54,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30424.62 MB 2025-02-14 19:16:54,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34556.01 MB 2025-02-14 19:16:54,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:16:54,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33646.71 MB 2025-02-14 19:16:54,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41668.31 MB 2025-02-14 19:16:54,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 19:16:54,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40100.29 MB 2025-02-14 19:16:54,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:16:54,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:16:54,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:16:54,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:54,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36089.55 MB 2025-02-14 19:16:54,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36856.56 MB 2025-02-14 19:16:54,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:16:54,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41668.31 MB 2025-02-14 19:16:54,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42083.55 MB 2025-02-14 19:16:54,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:16:54,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37564.34 MB 2025-02-14 19:16:54,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:16:54,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:16:54,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:16:54,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:54,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37269.45 MB 2025-02-14 19:16:54,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37497.52 MB 2025-02-14 19:16:54,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 19:16:54,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42083.55 MB 2025-02-14 19:16:54,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42083.55 MB 2025-02-14 19:16:54,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:16:54,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37728.83 MB 2025-02-14 19:16:54,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:16:54,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:16:54,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.10 seconds 2025-02-14 19:16:54,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:54,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22428.96 MB 2025-02-14 19:16:54,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37697.51 MB 2025-02-14 19:16:54,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15268.55 MB 2025-02-14 19:16:54,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48999.96 MB 2025-02-14 19:16:54,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42083.55 MB 2025-02-14 19:16:54,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6916.41 MB 2025-02-14 19:16:54,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37728.83 MB 2025-02-14 19:16:54,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:16:54,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:16:54,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:16:54,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:54,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37697.51 MB 2025-02-14 19:16:54,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27417.08 MB 2025-02-14 19:16:54,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10280.43 MB 2025-02-14 19:16:54,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42083.55 MB 2025-02-14 19:16:54,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42083.55 MB 2025-02-14 19:16:54,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:16:54,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40196.15 MB 2025-02-14 19:16:54,553 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 19:16:54,553 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:16:54,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:16:54,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:16:54,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:16:54,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:16:54,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27417.08 MB 2025-02-14 19:16:54,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35810.35 MB 2025-02-14 19:16:54,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 19:16:54,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42083.55 MB 2025-02-14 19:16:54,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46256.88 MB 2025-02-14 19:16:54,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 19:16:54,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35810.35 MB 2025-02-14 19:16:54,717 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 19:16:54,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:16:54,719 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:16:54,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:16:54,720 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:16:54,724 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:16:54,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:16:54,725 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:16:54,726 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:17:38,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:17:38,700 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:17:38,705 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:17:38,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:17:38,709 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1754, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:17:38,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:17:38,710 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1754, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:18:05,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:18:05,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:18:05,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.21 seconds 2025-02-14 19:18:05,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:05,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25190.86 MB 2025-02-14 19:18:05,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31398.43 MB 2025-02-14 19:18:05,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6207.57 MB 2025-02-14 19:18:05,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54603.55 MB 2025-02-14 19:18:05,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36838.57 MB 2025-02-14 19:18:05,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17764.97 MB 2025-02-14 19:18:05,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40325.35 MB 2025-02-14 19:18:06,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:18:06,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:18:06,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:18:06,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:06,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31398.43 MB 2025-02-14 19:18:06,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24896.35 MB 2025-02-14 19:18:06,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6502.08 MB 2025-02-14 19:18:06,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36838.57 MB 2025-02-14 19:18:06,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58271.47 MB 2025-02-14 19:18:06,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21432.89 MB 2025-02-14 19:18:06,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49243.43 MB 2025-02-14 19:18:07,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:18:07,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:18:07,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 19:18:07,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:07,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24896.35 MB 2025-02-14 19:18:07,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25427.19 MB 2025-02-14 19:18:07,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:18:07,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58271.47 MB 2025-02-14 19:18:07,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32046.58 MB 2025-02-14 19:18:07,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26224.89 MB 2025-02-14 19:18:07,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.74 MB 2025-02-14 19:18:08,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:18:08,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:18:08,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:18:08,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-14 19:18:08,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27316.72 MB 2025-02-14 19:18:08,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:18:08,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32046.58 MB 2025-02-14 19:18:08,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32048.68 MB 2025-02-14 19:18:08,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 19:18:08,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28734.15 MB 2025-02-14 19:18:08,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:18:08,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:18:08,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:18:08,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27316.72 MB 2025-02-14 19:18:08,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-14 19:18:08,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:18:08,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32048.68 MB 2025-02-14 19:18:08,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37239.13 MB 2025-02-14 19:18:08,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 19:18:08,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-14 19:18:08,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:18:08,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:18:08,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 19:18:08,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-14 19:18:08,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-14 19:18:08,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:18:08,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32046.58 MB 2025-02-14 19:18:08,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37239.13 MB 2025-02-14 19:18:08,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-14 19:18:08,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-14 19:18:08,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:18:08,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:18:08,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:18:08,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31092.12 MB 2025-02-14 19:18:08,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31859.12 MB 2025-02-14 19:18:08,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:18:08,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37239.13 MB 2025-02-14 19:18:08,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37656.46 MB 2025-02-14 19:18:08,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:18:08,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32566.91 MB 2025-02-14 19:18:08,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:18:08,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:18:08,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:18:08,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32272.01 MB 2025-02-14 19:18:08,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32501.00 MB 2025-02-14 19:18:08,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.99 MB 2025-02-14 19:18:08,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37656.46 MB 2025-02-14 19:18:08,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37656.46 MB 2025-02-14 19:18:08,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:08,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32711.87 MB 2025-02-14 19:18:08,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:18:08,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:18:08,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.70 seconds 2025-02-14 19:18:08,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19079.78 MB 2025-02-14 19:18:08,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32701.90 MB 2025-02-14 19:18:08,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13622.12 MB 2025-02-14 19:18:08,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54603.55 MB 2025-02-14 19:18:08,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37656.46 MB 2025-02-14 19:18:08,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16947.09 MB 2025-02-14 19:18:08,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32711.87 MB 2025-02-14 19:18:08,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:18:08,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:18:08,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:18:08,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32701.90 MB 2025-02-14 19:18:08,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24081.51 MB 2025-02-14 19:18:08,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8620.39 MB 2025-02-14 19:18:08,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37656.46 MB 2025-02-14 19:18:08,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37656.46 MB 2025-02-14 19:18:08,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:08,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35211.42 MB 2025-02-14 19:18:08,697 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 19:18:08,697 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:18:08,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:18:08,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:18:08,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:18:08,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:08,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24081.51 MB 2025-02-14 19:18:08,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32512.97 MB 2025-02-14 19:18:08,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 19:18:08,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37656.46 MB 2025-02-14 19:18:08,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46040.88 MB 2025-02-14 19:18:08,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 19:18:08,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32512.97 MB 2025-02-14 19:18:08,860 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 19:18:08,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:08,861 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:18:08,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:08,862 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:18:08,867 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:18:08,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:08,868 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:18:08,868 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:18:17,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:17,543 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:18:17,548 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:18:17,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:17,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1017, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:18:17,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:17,553 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1017, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:18:33,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:18:33,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:18:33,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.88 seconds 2025-02-14 19:18:33,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:33,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20055.33 MB 2025-02-14 19:18:33,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23654.43 MB 2025-02-14 19:18:33,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3599.11 MB 2025-02-14 19:18:33,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54425.29 MB 2025-02-14 19:18:33,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25887.24 MB 2025-02-14 19:18:33,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28538.04 MB 2025-02-14 19:18:33,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32471.90 MB 2025-02-14 19:18:33,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:18:33,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:18:33,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 19:18:33,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:33,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.43 MB 2025-02-14 19:18:33,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21065.97 MB 2025-02-14 19:18:33,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2588.47 MB 2025-02-14 19:18:33,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25887.24 MB 2025-02-14 19:18:33,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41372.61 MB 2025-02-14 19:18:33,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15485.37 MB 2025-02-14 19:18:33,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34171.01 MB 2025-02-14 19:18:35,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:18:35,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:18:35,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 19:18:35,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:35,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21065.97 MB 2025-02-14 19:18:35,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21596.81 MB 2025-02-14 19:18:35,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:18:35,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41372.61 MB 2025-02-14 19:18:35,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 19:18:35,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16252.93 MB 2025-02-14 19:18:35,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25576.39 MB 2025-02-14 19:18:35,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:18:35,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:18:35,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:18:35,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:35,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21596.81 MB 2025-02-14 19:18:35,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23486.34 MB 2025-02-14 19:18:35,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:18:35,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 19:18:35,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27007.12 MB 2025-02-14 19:18:35,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:18:35,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24903.77 MB 2025-02-14 19:18:35,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:18:35,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:18:35,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:18:35,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:35,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23486.34 MB 2025-02-14 19:18:35,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25728.20 MB 2025-02-14 19:18:35,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:18:35,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27007.12 MB 2025-02-14 19:18:35,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33141.29 MB 2025-02-14 19:18:35,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:18:35,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31272.48 MB 2025-02-14 19:18:35,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:18:35,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:18:35,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:18:35,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:35,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21596.81 MB 2025-02-14 19:18:35,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25728.20 MB 2025-02-14 19:18:35,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:18:35,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 19:18:35,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33141.29 MB 2025-02-14 19:18:35,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 19:18:35,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31272.48 MB 2025-02-14 19:18:35,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:18:35,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:18:35,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:18:35,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:35,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27261.74 MB 2025-02-14 19:18:35,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28028.74 MB 2025-02-14 19:18:35,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:18:35,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33141.29 MB 2025-02-14 19:18:35,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 19:18:35,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:18:35,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28736.53 MB 2025-02-14 19:18:35,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:18:35,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:18:35,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:18:35,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:35,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28441.63 MB 2025-02-14 19:18:35,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28668.31 MB 2025-02-14 19:18:35,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.68 MB 2025-02-14 19:18:35,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33558.63 MB 2025-02-14 19:18:35,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 19:18:35,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:35,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28844.95 MB 2025-02-14 19:18:35,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:18:35,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:18:35,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.32 seconds 2025-02-14 19:18:35,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:35,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16512.02 MB 2025-02-14 19:18:35,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28869.38 MB 2025-02-14 19:18:35,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12357.36 MB 2025-02-14 19:18:35,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54425.29 MB 2025-02-14 19:18:35,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 19:18:35,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20866.66 MB 2025-02-14 19:18:35,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28869.38 MB 2025-02-14 19:18:36,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:18:36,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:18:36,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:18:36,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:36,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28869.38 MB 2025-02-14 19:18:36,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21516.41 MB 2025-02-14 19:18:36,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7352.97 MB 2025-02-14 19:18:36,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33558.63 MB 2025-02-14 19:18:36,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 19:18:36,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:36,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31381.05 MB 2025-02-14 19:18:36,169 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:18:36,169 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:18:36,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:18:36,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:18:36,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:18:36,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:36,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21516.41 MB 2025-02-14 19:18:36,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29955.43 MB 2025-02-14 19:18:36,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:18:36,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33558.63 MB 2025-02-14 19:18:36,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41949.33 MB 2025-02-14 19:18:36,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:18:36,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29955.43 MB 2025-02-14 19:18:36,339 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:18:36,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:36,341 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:18:36,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:36,342 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:18:36,346 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:18:36,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:36,347 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:18:36,347 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:18:46,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:46,243 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:18:46,248 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:18:46,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:46,251 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:18:46,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:46,252 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:18:49,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:18:49,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:18:49,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.87 seconds 2025-02-14 19:18:49,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:49,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14250.85 MB 2025-02-14 19:18:49,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14902.01 MB 2025-02-14 19:18:49,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-14 19:18:49,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54534.34 MB 2025-02-14 19:18:49,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22162.70 MB 2025-02-14 19:18:49,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32371.64 MB 2025-02-14 19:18:49,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23722.22 MB 2025-02-14 19:18:49,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:18:49,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:18:49,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:18:49,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:49,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14902.01 MB 2025-02-14 19:18:49,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15147.21 MB 2025-02-14 19:18:49,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.19 MB 2025-02-14 19:18:49,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22162.70 MB 2025-02-14 19:18:49,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22162.70 MB 2025-02-14 19:18:49,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:49,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17374.34 MB 2025-02-14 19:18:49,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:18:49,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:18:49,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.83 seconds 2025-02-14 19:18:49,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:49,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15147.21 MB 2025-02-14 19:18:49,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15378.12 MB 2025-02-14 19:18:49,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.92 MB 2025-02-14 19:18:49,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22162.70 MB 2025-02-14 19:18:49,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22162.70 MB 2025-02-14 19:18:49,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:49,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19316.86 MB 2025-02-14 19:18:49,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:18:49,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:18:49,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:18:49,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:49,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15378.12 MB 2025-02-14 19:18:49,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16199.87 MB 2025-02-14 19:18:49,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 821.75 MB 2025-02-14 19:18:49,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22162.70 MB 2025-02-14 19:18:49,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22162.70 MB 2025-02-14 19:18:49,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:49,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16816.46 MB 2025-02-14 19:18:50,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:18:50,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:18:50,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:18:50,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:50,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16199.87 MB 2025-02-14 19:18:50,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17175.11 MB 2025-02-14 19:18:50,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 975.24 MB 2025-02-14 19:18:50,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22162.70 MB 2025-02-14 19:18:50,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22162.70 MB 2025-02-14 19:18:50,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:50,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19586.84 MB 2025-02-14 19:18:50,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:18:50,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:18:50,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 19:18:50,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:50,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15378.12 MB 2025-02-14 19:18:50,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17175.11 MB 2025-02-14 19:18:50,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1796.99 MB 2025-02-14 19:18:50,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22162.70 MB 2025-02-14 19:18:50,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22162.70 MB 2025-02-14 19:18:50,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:50,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19586.84 MB 2025-02-14 19:18:50,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:18:50,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:18:50,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 19:18:50,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:50,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17842.20 MB 2025-02-14 19:18:50,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18175.85 MB 2025-02-14 19:18:50,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 333.65 MB 2025-02-14 19:18:50,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22162.70 MB 2025-02-14 19:18:50,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22340.96 MB 2025-02-14 19:18:50,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 178.26 MB 2025-02-14 19:18:50,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18489.31 MB 2025-02-14 19:18:50,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:18:50,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:18:50,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:18:50,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:50,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18355.46 MB 2025-02-14 19:18:50,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18576.11 MB 2025-02-14 19:18:50,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.65 MB 2025-02-14 19:18:50,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22340.96 MB 2025-02-14 19:18:50,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22340.96 MB 2025-02-14 19:18:50,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:50,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18601.43 MB 2025-02-14 19:18:50,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:18:50,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:18:50,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.91 seconds 2025-02-14 19:18:50,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:50,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13609.78 MB 2025-02-14 19:18:50,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18777.16 MB 2025-02-14 19:18:50,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5167.38 MB 2025-02-14 19:18:50,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54534.34 MB 2025-02-14 19:18:50,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22340.96 MB 2025-02-14 19:18:50,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32193.38 MB 2025-02-14 19:18:50,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18777.16 MB 2025-02-14 19:18:50,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:18:50,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:18:50,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:18:50,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:50,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18777.16 MB 2025-02-14 19:18:50,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17546.90 MB 2025-02-14 19:18:50,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1230.26 MB 2025-02-14 19:18:50,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22340.96 MB 2025-02-14 19:18:50,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22340.96 MB 2025-02-14 19:18:50,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:18:50,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19011.55 MB 2025-02-14 19:18:50,453 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 19:18:50,453 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 19:18:50,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:18:50,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:18:50,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:18:50,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:18:50,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17546.90 MB 2025-02-14 19:18:50,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25985.73 MB 2025-02-14 19:18:50,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 19:18:50,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22340.96 MB 2025-02-14 19:18:50,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30729.57 MB 2025-02-14 19:18:50,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 19:18:50,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25985.73 MB 2025-02-14 19:18:50,621 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 19:18:50,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:50,623 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:18:50,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:50,624 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:18:50,628 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:18:50,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:18:50,629 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:18:50,629 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 19:19:48,314 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:19:48,314 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:19:48,319 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:19:48,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:19:48,323 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:19:48,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:19:48,324 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:19:51,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:19:51,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:19:51,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.85 seconds 2025-02-14 19:19:51,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:51,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-14 19:19:51,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-14 19:19:51,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-14 19:19:51,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39118.18 MB 2025-02-14 19:19:51,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 19:19:51,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16959.67 MB 2025-02-14 19:19:51,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.15 MB 2025-02-14 19:19:51,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:19:51,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:19:51,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:19:51,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:51,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-14 19:19:51,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15241.94 MB 2025-02-14 19:19:51,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.92 MB 2025-02-14 19:19:51,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22158.51 MB 2025-02-14 19:19:51,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 19:19:51,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:19:51,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17571.05 MB 2025-02-14 19:19:52,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:19:52,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:19:52,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.89 seconds 2025-02-14 19:19:52,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15241.94 MB 2025-02-14 19:19:52,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15488.79 MB 2025-02-14 19:19:52,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-14 19:19:52,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22158.51 MB 2025-02-14 19:19:52,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 19:19:52,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:19:52,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19411.59 MB 2025-02-14 19:19:52,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:19:52,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:19:52,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:19:52,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-14 19:19:52,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16367.14 MB 2025-02-14 19:19:52,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-14 19:19:52,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22158.51 MB 2025-02-14 19:19:52,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 19:19:52,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:19:52,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17026.25 MB 2025-02-14 19:19:52,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:19:52,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:19:52,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 19:19:52,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16367.14 MB 2025-02-14 19:19:52,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-14 19:19:52,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-14 19:19:52,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22158.51 MB 2025-02-14 19:19:52,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 19:19:52,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:19:52,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.70 MB 2025-02-14 19:19:52,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:19:52,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:19:52,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:19:52,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.72 MB 2025-02-14 19:19:52,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.64 MB 2025-02-14 19:19:52,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-14 19:19:52,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22158.51 MB 2025-02-14 19:19:52,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 19:19:52,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:19:52,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.70 MB 2025-02-14 19:19:52,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:19:52,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:19:52,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:19:52,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18122.98 MB 2025-02-14 19:19:52,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18479.63 MB 2025-02-14 19:19:52,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.66 MB 2025-02-14 19:19:52,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22158.51 MB 2025-02-14 19:19:52,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22353.54 MB 2025-02-14 19:19:52,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-14 19:19:52,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18812.52 MB 2025-02-14 19:19:52,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:19:52,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:19:52,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:19:52,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18671.63 MB 2025-02-14 19:19:52,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18876.52 MB 2025-02-14 19:19:52,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.88 MB 2025-02-14 19:19:52,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22353.54 MB 2025-02-14 19:19:52,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22357.74 MB 2025-02-14 19:19:52,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 19:19:52,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18902.86 MB 2025-02-14 19:19:52,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:19:52,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:19:52,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.96 seconds 2025-02-14 19:19:52,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-14 19:19:52,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19077.59 MB 2025-02-14 19:19:52,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5460.84 MB 2025-02-14 19:19:52,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39118.18 MB 2025-02-14 19:19:52,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22357.74 MB 2025-02-14 19:19:52,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16760.44 MB 2025-02-14 19:19:52,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.59 MB 2025-02-14 19:19:52,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:19:52,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:19:52,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 19:19:52,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.59 MB 2025-02-14 19:19:52,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17611.45 MB 2025-02-14 19:19:52,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1466.14 MB 2025-02-14 19:19:52,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22357.74 MB 2025-02-14 19:19:52,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22357.74 MB 2025-02-14 19:19:52,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:19:52,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19077.60 MB 2025-02-14 19:19:52,567 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:19:52,567 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 19:19:52,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:19:52,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:19:52,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:19:52,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:19:52,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17611.45 MB 2025-02-14 19:19:52,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26050.47 MB 2025-02-14 19:19:52,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:19:52,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22357.74 MB 2025-02-14 19:19:52,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30748.44 MB 2025-02-14 19:19:52,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:19:52,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26050.47 MB 2025-02-14 19:19:52,733 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:19:52,734 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:19:52,734 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:19:52,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:19:52,735 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:19:52,740 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:19:52,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:19:52,741 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:19:52,741 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 19:21:19,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:21:19,617 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:21:19,622 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:21:19,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:21:19,626 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1449, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:21:19,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:21:19,627 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1449, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:21:41,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:21:41,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:21:41,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.15 seconds 2025-02-14 19:21:41,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:41,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23065.57 MB 2025-02-14 19:21:41,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28193.50 MB 2025-02-14 19:21:41,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5127.93 MB 2025-02-14 19:21:41,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43333.45 MB 2025-02-14 19:21:41,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34496.05 MB 2025-02-14 19:21:41,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8837.40 MB 2025-02-14 19:21:41,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37066.79 MB 2025-02-14 19:21:41,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:21:41,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:21:41,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:21:41,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:41,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28193.50 MB 2025-02-14 19:21:41,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23310.75 MB 2025-02-14 19:21:41,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4882.75 MB 2025-02-14 19:21:41,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34496.05 MB 2025-02-14 19:21:41,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45009.08 MB 2025-02-14 19:21:41,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10513.02 MB 2025-02-14 19:21:41,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40021.87 MB 2025-02-14 19:21:43,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:21:43,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:21:43,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 19:21:43,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:43,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23310.75 MB 2025-02-14 19:21:43,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23841.59 MB 2025-02-14 19:21:43,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:21:43,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45009.08 MB 2025-02-14 19:21:43,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27915.19 MB 2025-02-14 19:21:43,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17093.89 MB 2025-02-14 19:21:43,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27821.18 MB 2025-02-14 19:21:43,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:21:43,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:21:43,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:21:43,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:43,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23841.59 MB 2025-02-14 19:21:43,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25731.13 MB 2025-02-14 19:21:43,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:21:43,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27915.19 MB 2025-02-14 19:21:43,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29802.63 MB 2025-02-14 19:21:43,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:21:43,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27148.55 MB 2025-02-14 19:21:43,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:21:43,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:21:43,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:21:43,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:43,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25731.13 MB 2025-02-14 19:21:43,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27972.98 MB 2025-02-14 19:21:43,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:21:43,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29802.63 MB 2025-02-14 19:21:43,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35467.03 MB 2025-02-14 19:21:43,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 19:21:43,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33517.26 MB 2025-02-14 19:21:43,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:21:43,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:21:43,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:21:43,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:43,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23841.59 MB 2025-02-14 19:21:43,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27972.98 MB 2025-02-14 19:21:43,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:21:43,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27915.19 MB 2025-02-14 19:21:43,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35467.03 MB 2025-02-14 19:21:43,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7551.84 MB 2025-02-14 19:21:43,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33517.26 MB 2025-02-14 19:21:44,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:21:44,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:21:44,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:21:44,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:44,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29506.52 MB 2025-02-14 19:21:44,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30273.53 MB 2025-02-14 19:21:44,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:21:44,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35467.03 MB 2025-02-14 19:21:44,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35884.37 MB 2025-02-14 19:21:44,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:21:44,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30981.31 MB 2025-02-14 19:21:44,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:21:44,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:21:44,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:21:44,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:44,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30686.41 MB 2025-02-14 19:21:44,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30914.71 MB 2025-02-14 19:21:44,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-14 19:21:44,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35884.37 MB 2025-02-14 19:21:44,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35884.37 MB 2025-02-14 19:21:44,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:21:44,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31138.36 MB 2025-02-14 19:21:44,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:21:44,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:21:44,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.55 seconds 2025-02-14 19:21:44,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:44,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18017.14 MB 2025-02-14 19:21:44,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31115.19 MB 2025-02-14 19:21:44,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13098.06 MB 2025-02-14 19:21:44,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43333.45 MB 2025-02-14 19:21:44,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35884.37 MB 2025-02-14 19:21:44,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7449.08 MB 2025-02-14 19:21:44,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31138.36 MB 2025-02-14 19:21:44,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:21:44,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:21:44,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:21:44,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:44,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31115.19 MB 2025-02-14 19:21:44,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23008.47 MB 2025-02-14 19:21:44,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8106.73 MB 2025-02-14 19:21:44,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35884.37 MB 2025-02-14 19:21:44,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35884.37 MB 2025-02-14 19:21:44,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:21:44,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33616.11 MB 2025-02-14 19:21:44,476 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 19:21:44,477 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:21:44,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:21:44,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:21:44,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:21:44,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:21:44,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23008.47 MB 2025-02-14 19:21:44,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31412.03 MB 2025-02-14 19:21:44,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 19:21:44,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35884.37 MB 2025-02-14 19:21:44,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44239.42 MB 2025-02-14 19:21:44,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 19:21:44,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31412.03 MB 2025-02-14 19:21:44,720 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 19:21:44,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:21:44,722 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:21:44,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:21:44,724 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:21:44,731 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:21:44,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:21:44,733 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:21:44,733 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:24:11,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:24:11,277 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:24:11,283 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:24:11,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:24:11,287 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2330, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:24:11,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:24:11,288 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2330, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:24:47,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:24:47,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:24:47,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.71 seconds 2025-02-14 19:24:47,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:47,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29204.52 MB 2025-02-14 19:24:47,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37450.52 MB 2025-02-14 19:24:47,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8246.00 MB 2025-02-14 19:24:47,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52594.48 MB 2025-02-14 19:24:47,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38889.59 MB 2025-02-14 19:24:47,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13704.89 MB 2025-02-14 19:24:47,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46377.44 MB 2025-02-14 19:24:47,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:24:47,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:24:47,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:24:47,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:47,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37450.52 MB 2025-02-14 19:24:47,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27891.84 MB 2025-02-14 19:24:47,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9558.68 MB 2025-02-14 19:24:47,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38889.59 MB 2025-02-14 19:24:47,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62669.19 MB 2025-02-14 19:24:47,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23779.61 MB 2025-02-14 19:24:47,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52433.11 MB 2025-02-14 19:24:49,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:24:49,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:24:49,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.00 seconds 2025-02-14 19:24:49,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27891.84 MB 2025-02-14 19:24:49,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28422.68 MB 2025-02-14 19:24:49,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:24:49,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62669.19 MB 2025-02-14 19:24:49,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29831.99 MB 2025-02-14 19:24:49,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32837.21 MB 2025-02-14 19:24:49,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32403.31 MB 2025-02-14 19:24:49,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:24:49,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:24:49,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:24:49,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28422.68 MB 2025-02-14 19:24:49,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30311.96 MB 2025-02-14 19:24:49,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-14 19:24:49,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29831.99 MB 2025-02-14 19:24:49,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33135.00 MB 2025-02-14 19:24:49,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 19:24:49,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31729.38 MB 2025-02-14 19:24:49,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:24:49,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:24:49,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:24:49,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30311.96 MB 2025-02-14 19:24:49,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32553.81 MB 2025-02-14 19:24:49,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:24:49,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33135.00 MB 2025-02-14 19:24:49,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39743.13 MB 2025-02-14 19:24:49,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-14 19:24:49,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38098.09 MB 2025-02-14 19:24:49,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:24:49,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:24:49,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:24:49,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28422.68 MB 2025-02-14 19:24:49,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32553.81 MB 2025-02-14 19:24:49,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-14 19:24:49,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29831.99 MB 2025-02-14 19:24:49,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39743.13 MB 2025-02-14 19:24:49,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9911.14 MB 2025-02-14 19:24:49,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38098.09 MB 2025-02-14 19:24:49,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:24:49,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:24:49,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:24:49,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34087.35 MB 2025-02-14 19:24:49,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34854.36 MB 2025-02-14 19:24:49,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:24:49,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39743.13 MB 2025-02-14 19:24:49,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40160.46 MB 2025-02-14 19:24:49,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:24:49,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35562.14 MB 2025-02-14 19:24:49,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:24:49,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:24:49,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:24:49,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35267.24 MB 2025-02-14 19:24:49,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35496.46 MB 2025-02-14 19:24:49,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.22 MB 2025-02-14 19:24:49,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40160.46 MB 2025-02-14 19:24:49,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40160.46 MB 2025-02-14 19:24:49,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:24:49,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35732.52 MB 2025-02-14 19:24:49,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:24:49,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:24:49,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.25 seconds 2025-02-14 19:24:49,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21086.61 MB 2025-02-14 19:24:49,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35697.54 MB 2025-02-14 19:24:49,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14610.92 MB 2025-02-14 19:24:49,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52594.48 MB 2025-02-14 19:24:49,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40160.46 MB 2025-02-14 19:24:49,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12434.01 MB 2025-02-14 19:24:49,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35732.52 MB 2025-02-14 19:24:49,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:24:49,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:24:49,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:24:49,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35697.54 MB 2025-02-14 19:24:49,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26091.00 MB 2025-02-14 19:24:49,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9606.53 MB 2025-02-14 19:24:49,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40160.46 MB 2025-02-14 19:24:49,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40160.46 MB 2025-02-14 19:24:49,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:24:49,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38209.20 MB 2025-02-14 19:24:49,825 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:24:49,826 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 19:24:49,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:24:49,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:24:49,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:24:49,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:24:49,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26091.00 MB 2025-02-14 19:24:49,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34530.02 MB 2025-02-14 19:24:49,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:24:49,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40160.46 MB 2025-02-14 19:24:49,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48551.17 MB 2025-02-14 19:24:49,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:24:49,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34530.02 MB 2025-02-14 19:24:49,989 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:24:49,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:24:49,991 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:24:49,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:24:49,992 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:24:49,996 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:24:49,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:24:49,997 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:24:49,997 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 19:26:41,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:26:41,553 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:26:41,558 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:26:41,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:26:41,562 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3112, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:26:41,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:26:41,563 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3112, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:27:29,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:27:29,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:27:29,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.93 seconds 2025-02-14 19:27:29,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:29,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34654.67 MB 2025-02-14 19:27:29,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45668.91 MB 2025-02-14 19:27:29,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11014.24 MB 2025-02-14 19:27:29,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82824.92 MB 2025-02-14 19:27:29,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50816.09 MB 2025-02-14 19:27:29,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32008.83 MB 2025-02-14 19:27:29,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56682.10 MB 2025-02-14 19:27:29,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:27:29,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:27:29,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:27:29,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:29,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45668.91 MB 2025-02-14 19:27:29,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31956.69 MB 2025-02-14 19:27:29,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -13712.22 MB 2025-02-14 19:27:29,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50816.09 MB 2025-02-14 19:27:29,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 89877.64 MB 2025-02-14 19:27:29,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 39061.55 MB 2025-02-14 19:27:29,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 77420.83 MB 2025-02-14 19:27:31,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:27:31,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:27:31,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 19:27:31,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:31,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31956.69 MB 2025-02-14 19:27:31,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32487.53 MB 2025-02-14 19:27:31,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:27:31,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 89877.64 MB 2025-02-14 19:27:31,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35710.30 MB 2025-02-14 19:27:31,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -54167.34 MB 2025-02-14 19:27:31,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36467.12 MB 2025-02-14 19:27:31,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:27:31,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:27:31,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:27:31,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:31,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32487.53 MB 2025-02-14 19:27:31,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34377.07 MB 2025-02-14 19:27:31,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:27:31,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35710.30 MB 2025-02-14 19:27:31,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37597.74 MB 2025-02-14 19:27:31,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:27:31,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35794.50 MB 2025-02-14 19:27:31,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:27:31,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:27:31,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:27:31,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:31,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34377.07 MB 2025-02-14 19:27:31,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36618.92 MB 2025-02-14 19:27:31,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:27:31,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37597.74 MB 2025-02-14 19:27:31,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43731.91 MB 2025-02-14 19:27:31,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:27:31,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42163.21 MB 2025-02-14 19:27:31,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:27:31,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:27:31,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:27:31,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:31,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32487.53 MB 2025-02-14 19:27:31,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36618.92 MB 2025-02-14 19:27:31,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:27:31,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35710.30 MB 2025-02-14 19:27:31,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43731.91 MB 2025-02-14 19:27:31,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 19:27:31,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42163.21 MB 2025-02-14 19:27:32,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:27:32,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:27:32,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:27:32,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:32,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38152.47 MB 2025-02-14 19:27:32,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38919.47 MB 2025-02-14 19:27:32,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:27:32,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43731.91 MB 2025-02-14 19:27:32,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44149.24 MB 2025-02-14 19:27:32,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:27:32,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39627.26 MB 2025-02-14 19:27:32,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:27:32,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:27:32,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:27:32,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:32,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39332.36 MB 2025-02-14 19:27:32,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39561.99 MB 2025-02-14 19:27:32,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.64 MB 2025-02-14 19:27:32,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44149.24 MB 2025-02-14 19:27:32,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44149.24 MB 2025-02-14 19:27:32,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:27:32,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39787.22 MB 2025-02-14 19:27:32,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:27:32,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:27:32,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 50.54 seconds 2025-02-14 19:27:32,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:32,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23811.69 MB 2025-02-14 19:27:32,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39763.07 MB 2025-02-14 19:27:32,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15951.38 MB 2025-02-14 19:27:32,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71980.55 MB 2025-02-14 19:27:32,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44149.24 MB 2025-02-14 19:27:32,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27831.30 MB 2025-02-14 19:27:32,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39787.22 MB 2025-02-14 19:27:32,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:27:32,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:27:32,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:27:32,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:32,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39763.07 MB 2025-02-14 19:27:32,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28816.08 MB 2025-02-14 19:27:32,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10946.99 MB 2025-02-14 19:27:32,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44149.24 MB 2025-02-14 19:27:32,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44149.24 MB 2025-02-14 19:27:32,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:27:32,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42274.73 MB 2025-02-14 19:27:32,399 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:27:32,400 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 19:27:32,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:27:32,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:27:32,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:27:32,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:27:32,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28816.08 MB 2025-02-14 19:27:32,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37255.10 MB 2025-02-14 19:27:32,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:27:32,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44149.24 MB 2025-02-14 19:27:32,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48343.55 MB 2025-02-14 19:27:32,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 19:27:32,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37255.10 MB 2025-02-14 19:27:32,565 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:27:32,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:27:32,566 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:27:32,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:27:32,567 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:27:32,572 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:27:32,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:27:32,573 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:27:32,573 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 19:27:43,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:27:43,198 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:27:43,207 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:27:43,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:27:43,215 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2451, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:27:43,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:27:43,217 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2451, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:28:21,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:28:21,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:28:21,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.37 seconds 2025-02-14 19:28:21,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:21,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30047.86 MB 2025-02-14 19:28:21,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38721.81 MB 2025-02-14 19:28:21,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8673.95 MB 2025-02-14 19:28:21,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69470.26 MB 2025-02-14 19:28:21,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46005.22 MB 2025-02-14 19:28:21,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23465.03 MB 2025-02-14 19:28:21,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47672.95 MB 2025-02-14 19:28:21,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:28:21,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:28:21,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 19:28:21,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:21,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38721.81 MB 2025-02-14 19:28:21,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28520.02 MB 2025-02-14 19:28:21,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10201.78 MB 2025-02-14 19:28:21,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46005.22 MB 2025-02-14 19:28:21,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71116.52 MB 2025-02-14 19:28:21,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25111.30 MB 2025-02-14 19:28:21,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62774.64 MB 2025-02-14 19:28:23,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:28:23,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:28:23,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 19:28:23,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:23,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28520.02 MB 2025-02-14 19:28:23,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29050.87 MB 2025-02-14 19:28:23,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:28:23,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71116.52 MB 2025-02-14 19:28:23,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37329.31 MB 2025-02-14 19:28:23,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33787.22 MB 2025-02-14 19:28:23,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33029.41 MB 2025-02-14 19:28:23,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:28:23,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:28:23,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:28:23,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:23,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29050.87 MB 2025-02-14 19:28:23,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30940.33 MB 2025-02-14 19:28:23,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.47 MB 2025-02-14 19:28:23,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37329.31 MB 2025-02-14 19:28:23,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37329.31 MB 2025-02-14 19:28:23,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:23,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32357.76 MB 2025-02-14 19:28:23,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:28:23,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:28:23,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:28:23,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:23,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30940.33 MB 2025-02-14 19:28:23,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33182.19 MB 2025-02-14 19:28:23,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:28:23,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37329.31 MB 2025-02-14 19:28:23,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41104.18 MB 2025-02-14 19:28:23,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 19:28:23,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38726.47 MB 2025-02-14 19:28:23,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:28:23,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:28:23,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 19:28:23,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:23,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29050.87 MB 2025-02-14 19:28:23,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33182.19 MB 2025-02-14 19:28:23,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-14 19:28:23,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37329.31 MB 2025-02-14 19:28:23,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41104.18 MB 2025-02-14 19:28:23,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 19:28:23,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38726.47 MB 2025-02-14 19:28:24,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:28:24,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:28:24,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:28:24,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:24,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34715.73 MB 2025-02-14 19:28:24,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35482.73 MB 2025-02-14 19:28:24,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:28:24,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41104.18 MB 2025-02-14 19:28:24,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41521.51 MB 2025-02-14 19:28:24,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:28:24,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36190.52 MB 2025-02-14 19:28:24,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:28:24,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:28:24,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:28:24,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:24,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35895.62 MB 2025-02-14 19:28:24,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36123.64 MB 2025-02-14 19:28:24,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.01 MB 2025-02-14 19:28:24,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41521.51 MB 2025-02-14 19:28:24,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41521.51 MB 2025-02-14 19:28:24,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:24,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36342.48 MB 2025-02-14 19:28:24,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:28:24,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:28:24,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.92 seconds 2025-02-14 19:28:24,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:24,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21508.37 MB 2025-02-14 19:28:24,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36323.80 MB 2025-02-14 19:28:24,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14815.43 MB 2025-02-14 19:28:24,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69470.26 MB 2025-02-14 19:28:24,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41521.51 MB 2025-02-14 19:28:24,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27948.74 MB 2025-02-14 19:28:24,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36342.48 MB 2025-02-14 19:28:24,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:28:24,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:28:24,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:28:24,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:24,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36323.80 MB 2025-02-14 19:28:24,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26498.99 MB 2025-02-14 19:28:24,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9824.81 MB 2025-02-14 19:28:24,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41521.51 MB 2025-02-14 19:28:24,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41521.51 MB 2025-02-14 19:28:24,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:24,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38824.10 MB 2025-02-14 19:28:24,431 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 19:28:24,432 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:28:24,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:28:24,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:28:24,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:28:24,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:24,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.99 MB 2025-02-14 19:28:24,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34899.49 MB 2025-02-14 19:28:24,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.50 MB 2025-02-14 19:28:24,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41521.51 MB 2025-02-14 19:28:24,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41521.51 MB 2025-02-14 19:28:24,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:24,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34899.49 MB 2025-02-14 19:28:24,602 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 19:28:24,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:24,603 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:28:24,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:24,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:28:24,609 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:28:24,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:24,610 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:28:24,610 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:28:50,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:50,482 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:28:50,487 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:28:50,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:50,491 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 155, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:28:50,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:50,492 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 155, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:28:52,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:28:52,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:28:52,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-14 19:28:52,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:52,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14048.77 MB 2025-02-14 19:28:52,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14597.31 MB 2025-02-14 19:28:52,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 548.54 MB 2025-02-14 19:28:52,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49872.37 MB 2025-02-14 19:28:52,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 19:28:52,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29869.74 MB 2025-02-14 19:28:52,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23520.14 MB 2025-02-14 19:28:52,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:28:52,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:28:52,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:28:52,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:52,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14597.31 MB 2025-02-14 19:28:52,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14820.93 MB 2025-02-14 19:28:52,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.63 MB 2025-02-14 19:28:52,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 19:28:52,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 19:28:52,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:52,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16729.16 MB 2025-02-14 19:28:53,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:28:53,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:28:53,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 19:28:53,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:53,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14820.93 MB 2025-02-14 19:28:53,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15018.67 MB 2025-02-14 19:28:53,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.74 MB 2025-02-14 19:28:53,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 19:28:53,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 19:28:53,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:53,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18990.58 MB 2025-02-14 19:28:53,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:28:53,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:28:53,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:28:53,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:53,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15018.61 MB 2025-02-14 19:28:53,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15722.29 MB 2025-02-14 19:28:53,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 703.68 MB 2025-02-14 19:28:53,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 19:28:53,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 19:28:53,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:53,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16250.29 MB 2025-02-14 19:28:53,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:28:53,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:28:53,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:28:53,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:53,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15722.29 MB 2025-02-14 19:28:53,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16557.78 MB 2025-02-14 19:28:53,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.49 MB 2025-02-14 19:28:53,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 19:28:53,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 19:28:53,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:53,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18622.98 MB 2025-02-14 19:28:53,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:28:53,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:28:53,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:28:53,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:53,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15018.61 MB 2025-02-14 19:28:53,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16557.78 MB 2025-02-14 19:28:53,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1539.17 MB 2025-02-14 19:28:53,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 19:28:53,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 19:28:53,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:53,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18622.98 MB 2025-02-14 19:28:53,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:28:53,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:28:53,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:28:53,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:53,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17129.02 MB 2025-02-14 19:28:53,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17414.73 MB 2025-02-14 19:28:53,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 285.71 MB 2025-02-14 19:28:53,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 19:28:53,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20155.73 MB 2025-02-14 19:28:53,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-14 19:28:53,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17687.52 MB 2025-02-14 19:28:53,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:28:53,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:28:53,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:28:53,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:53,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17568.54 MB 2025-02-14 19:28:53,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17784.59 MB 2025-02-14 19:28:53,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.05 MB 2025-02-14 19:28:53,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20155.73 MB 2025-02-14 19:28:53,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20155.73 MB 2025-02-14 19:28:53,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:53,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17798.66 MB 2025-02-14 19:28:53,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:28:53,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:28:53,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.32 seconds 2025-02-14 19:28:53,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:53,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13508.74 MB 2025-02-14 19:28:53,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17985.36 MB 2025-02-14 19:28:53,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4476.63 MB 2025-02-14 19:28:53,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49872.37 MB 2025-02-14 19:28:53,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20155.73 MB 2025-02-14 19:28:53,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29716.64 MB 2025-02-14 19:28:53,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17985.36 MB 2025-02-14 19:28:54,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:28:54,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:28:54,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 19:28:54,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:54,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17985.36 MB 2025-02-14 19:28:54,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17324.01 MB 2025-02-14 19:28:54,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -661.35 MB 2025-02-14 19:28:54,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20155.73 MB 2025-02-14 19:28:54,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20155.73 MB 2025-02-14 19:28:54,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:28:54,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18988.56 MB 2025-02-14 19:28:54,097 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 19:28:54,097 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:28:54,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:28:54,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:28:54,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:28:54,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:28:54,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17324.01 MB 2025-02-14 19:28:54,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25750.52 MB 2025-02-14 19:28:54,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 19:28:54,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20155.73 MB 2025-02-14 19:28:54,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30628.90 MB 2025-02-14 19:28:54,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 19:28:54,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25750.52 MB 2025-02-14 19:28:54,259 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 19:28:54,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:54,261 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:28:54,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:54,262 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:28:54,266 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:28:54,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:28:54,267 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:28:54,267 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:29:53,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:29:53,686 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:29:53,691 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:29:53,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:29:53,695 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 506, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:29:53,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:29:53,696 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 506, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:30:01,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:30:01,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:30:01,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.75 seconds 2025-02-14 19:30:01,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:01,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16494.60 MB 2025-02-14 19:30:01,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18285.30 MB 2025-02-14 19:30:01,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1790.71 MB 2025-02-14 19:30:01,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43195.04 MB 2025-02-14 19:30:01,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22097.69 MB 2025-02-14 19:30:01,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21097.35 MB 2025-02-14 19:30:01,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27098.43 MB 2025-02-14 19:30:01,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:30:01,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:30:01,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 19:30:01,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:01,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18285.30 MB 2025-02-14 19:30:01,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18409.44 MB 2025-02-14 19:30:01,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 124.13 MB 2025-02-14 19:30:01,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22097.69 MB 2025-02-14 19:30:01,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29131.54 MB 2025-02-14 19:30:01,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7033.85 MB 2025-02-14 19:30:01,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26037.62 MB 2025-02-14 19:30:03,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:30:03,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:30:03,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 19:30:03,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:03,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18409.44 MB 2025-02-14 19:30:03,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18940.28 MB 2025-02-14 19:30:03,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:30:03,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29131.54 MB 2025-02-14 19:30:03,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 19:30:03,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5853.15 MB 2025-02-14 19:30:03,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22918.82 MB 2025-02-14 19:30:03,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:30:03,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:30:03,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:30:03,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:03,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18940.28 MB 2025-02-14 19:30:03,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20829.81 MB 2025-02-14 19:30:03,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:30:03,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 19:30:03,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24222.11 MB 2025-02-14 19:30:03,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 19:30:03,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22247.24 MB 2025-02-14 19:30:03,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:30:03,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:30:03,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:30:03,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:03,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20829.81 MB 2025-02-14 19:30:03,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23071.67 MB 2025-02-14 19:30:03,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:30:03,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24222.11 MB 2025-02-14 19:30:03,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 19:30:03,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:30:03,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28615.95 MB 2025-02-14 19:30:03,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:30:03,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:30:03,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:30:03,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:03,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18940.28 MB 2025-02-14 19:30:03,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23071.67 MB 2025-02-14 19:30:03,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:30:03,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 19:30:03,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 19:30:03,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 19:30:03,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28615.95 MB 2025-02-14 19:30:03,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:30:03,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:30:03,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:30:03,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:03,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24605.21 MB 2025-02-14 19:30:03,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25372.21 MB 2025-02-14 19:30:03,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:30:03,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30356.28 MB 2025-02-14 19:30:03,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 19:30:03,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:30:03,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26080.00 MB 2025-02-14 19:30:03,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:30:03,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:30:03,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:30:03,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:03,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25785.10 MB 2025-02-14 19:30:03,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26012.54 MB 2025-02-14 19:30:03,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.44 MB 2025-02-14 19:30:03,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30771.51 MB 2025-02-14 19:30:03,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 19:30:03,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:30:03,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26240.78 MB 2025-02-14 19:30:03,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:30:03,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:30:03,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.10 seconds 2025-02-14 19:30:03,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:03,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14731.65 MB 2025-02-14 19:30:03,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26213.02 MB 2025-02-14 19:30:03,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11481.37 MB 2025-02-14 19:30:03,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43195.04 MB 2025-02-14 19:30:03,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 19:30:03,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12423.53 MB 2025-02-14 19:30:03,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26240.78 MB 2025-02-14 19:30:04,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:30:04,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:30:04,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:30:04,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:04,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26213.02 MB 2025-02-14 19:30:04,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19726.90 MB 2025-02-14 19:30:04,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6486.12 MB 2025-02-14 19:30:04,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30771.51 MB 2025-02-14 19:30:04,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 19:30:04,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:30:04,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28717.31 MB 2025-02-14 19:30:04,083 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 19:30:04,084 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:30:04,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:30:04,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:30:04,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:30:04,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:30:04,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19726.90 MB 2025-02-14 19:30:04,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28140.88 MB 2025-02-14 19:30:04,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 19:30:04,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30771.51 MB 2025-02-14 19:30:04,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39137.05 MB 2025-02-14 19:30:04,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-14 19:30:04,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28140.88 MB 2025-02-14 19:30:04,247 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 19:30:04,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:30:04,248 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:30:04,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:30:04,249 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:30:04,254 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:30:04,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:30:04,255 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:30:04,255 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:31:18,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:31:18,437 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:31:18,442 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:31:18,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:31:18,446 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1552, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:31:18,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:31:18,447 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1552, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:31:42,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:31:42,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:31:42,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.88 seconds 2025-02-14 19:31:42,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:42,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23783.29 MB 2025-02-14 19:31:42,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.73 MB 2025-02-14 19:31:42,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5492.44 MB 2025-02-14 19:31:42,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51684.31 MB 2025-02-14 19:31:42,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36152.80 MB 2025-02-14 19:31:42,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15531.51 MB 2025-02-14 19:31:42,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38238.30 MB 2025-02-14 19:31:42,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:31:42,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:31:42,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:31:42,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:42,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29275.73 MB 2025-02-14 19:31:42,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23846.21 MB 2025-02-14 19:31:42,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5429.52 MB 2025-02-14 19:31:42,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36152.80 MB 2025-02-14 19:31:42,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49297.75 MB 2025-02-14 19:31:42,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13144.95 MB 2025-02-14 19:31:42,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44193.94 MB 2025-02-14 19:31:44,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:31:44,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:31:44,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 19:31:44,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:44,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23846.21 MB 2025-02-14 19:31:44,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24377.06 MB 2025-02-14 19:31:44,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:31:44,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49297.75 MB 2025-02-14 19:31:44,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 19:31:44,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18637.39 MB 2025-02-14 19:31:44,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28355.60 MB 2025-02-14 19:31:44,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:31:44,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:31:44,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:31:44,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:44,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24377.06 MB 2025-02-14 19:31:44,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26266.59 MB 2025-02-14 19:31:44,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:31:44,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 19:31:44,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 19:31:44,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:31:44,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27684.02 MB 2025-02-14 19:31:44,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:31:44,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:31:44,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:31:44,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:44,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26266.59 MB 2025-02-14 19:31:44,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28508.45 MB 2025-02-14 19:31:44,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:31:44,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 19:31:44,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 19:31:44,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:31:44,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34052.73 MB 2025-02-14 19:31:44,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:31:44,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:31:44,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 19:31:44,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:44,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24377.06 MB 2025-02-14 19:31:44,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28508.45 MB 2025-02-14 19:31:44,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:31:44,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 19:31:44,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 19:31:44,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:31:44,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34052.73 MB 2025-02-14 19:31:44,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:31:44,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:31:44,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:31:44,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:44,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30041.99 MB 2025-02-14 19:31:44,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30808.99 MB 2025-02-14 19:31:44,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:31:44,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36322.67 MB 2025-02-14 19:31:44,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 19:31:44,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:31:44,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31516.78 MB 2025-02-14 19:31:44,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:31:44,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:31:44,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:31:44,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:44,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31221.88 MB 2025-02-14 19:31:44,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31452.46 MB 2025-02-14 19:31:44,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.58 MB 2025-02-14 19:31:44,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36737.91 MB 2025-02-14 19:31:44,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 19:31:44,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:31:44,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31652.42 MB 2025-02-14 19:31:44,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:31:44,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:31:44,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.33 seconds 2025-02-14 19:31:44,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:44,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18376.00 MB 2025-02-14 19:31:44,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31653.54 MB 2025-02-14 19:31:44,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13277.54 MB 2025-02-14 19:31:44,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51684.31 MB 2025-02-14 19:31:44,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 19:31:44,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14946.40 MB 2025-02-14 19:31:44,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31653.54 MB 2025-02-14 19:31:45,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:31:45,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:31:45,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:31:45,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:45,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31653.54 MB 2025-02-14 19:31:45,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23380.39 MB 2025-02-14 19:31:45,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8273.15 MB 2025-02-14 19:31:45,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36737.91 MB 2025-02-14 19:31:45,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 19:31:45,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:31:45,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34165.20 MB 2025-02-14 19:31:45,062 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:31:45,063 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:31:45,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:31:45,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:31:45,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:31:45,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:31:45,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23380.39 MB 2025-02-14 19:31:45,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31819.41 MB 2025-02-14 19:31:45,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:31:45,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36737.91 MB 2025-02-14 19:31:45,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45128.61 MB 2025-02-14 19:31:45,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:31:45,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31819.41 MB 2025-02-14 19:31:45,269 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:31:45,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:31:45,272 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:31:45,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:31:45,274 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:31:45,281 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:31:45,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:31:45,283 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:31:45,284 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:33:02,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:33:02,452 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:33:02,457 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:33:02,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:33:02,460 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1801, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:33:02,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:33:02,461 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1801, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:33:30,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:33:30,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:33:30,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.68 seconds 2025-02-14 19:33:30,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:30,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25518.36 MB 2025-02-14 19:33:30,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31892.00 MB 2025-02-14 19:33:30,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6373.64 MB 2025-02-14 19:33:30,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57713.62 MB 2025-02-14 19:33:30,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37052.48 MB 2025-02-14 19:33:30,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20661.14 MB 2025-02-14 19:33:30,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40879.34 MB 2025-02-14 19:33:30,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:33:30,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:33:30,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:33:30,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:30,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31892.00 MB 2025-02-14 19:33:30,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25140.69 MB 2025-02-14 19:33:30,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6751.31 MB 2025-02-14 19:33:30,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37052.48 MB 2025-02-14 19:33:30,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59175.34 MB 2025-02-14 19:33:30,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22122.86 MB 2025-02-14 19:33:30,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49999.55 MB 2025-02-14 19:33:32,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:33:32,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:33:32,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 19:33:32,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25140.69 MB 2025-02-14 19:33:32,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.53 MB 2025-02-14 19:33:32,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:33:32,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59175.34 MB 2025-02-14 19:33:32,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32092.72 MB 2025-02-14 19:33:32,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27082.62 MB 2025-02-14 19:33:32,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29650.08 MB 2025-02-14 19:33:32,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:33:32,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:33:32,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:33:32,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-14 19:33:32,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27561.06 MB 2025-02-14 19:33:32,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:33:32,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 19:33:32,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32092.72 MB 2025-02-14 19:33:32,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:33:32,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28978.49 MB 2025-02-14 19:33:32,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:33:32,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:33:32,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:33:32,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27561.06 MB 2025-02-14 19:33:32,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-14 19:33:32,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:33:32,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 19:33:32,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37755.03 MB 2025-02-14 19:33:32,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:33:32,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-14 19:33:32,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:33:32,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:33:32,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:33:32,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-14 19:33:32,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-14 19:33:32,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:33:32,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 19:33:32,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37755.03 MB 2025-02-14 19:33:32,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:33:32,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-14 19:33:32,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:33:32,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:33:32,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:33:32,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31336.46 MB 2025-02-14 19:33:32,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32103.46 MB 2025-02-14 19:33:32,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:33:32,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37755.03 MB 2025-02-14 19:33:32,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 19:33:32,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 19:33:32,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32811.25 MB 2025-02-14 19:33:32,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:33:32,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:33:32,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:33:32,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32516.35 MB 2025-02-14 19:33:32,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32745.33 MB 2025-02-14 19:33:32,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.98 MB 2025-02-14 19:33:32,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38166.07 MB 2025-02-14 19:33:32,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 19:33:32,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:33:32,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32938.57 MB 2025-02-14 19:33:32,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:33:32,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:33:32,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.14 seconds 2025-02-14 19:33:32,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19243.53 MB 2025-02-14 19:33:32,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32946.40 MB 2025-02-14 19:33:32,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13702.87 MB 2025-02-14 19:33:32,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57713.62 MB 2025-02-14 19:33:32,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 19:33:32,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19547.55 MB 2025-02-14 19:33:32,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32946.40 MB 2025-02-14 19:33:32,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:33:32,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:33:32,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:33:32,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32946.40 MB 2025-02-14 19:33:32,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24247.60 MB 2025-02-14 19:33:32,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8698.81 MB 2025-02-14 19:33:32,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38166.07 MB 2025-02-14 19:33:32,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 19:33:32,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:33:32,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35458.07 MB 2025-02-14 19:33:32,891 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:33:32,891 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:33:32,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:33:32,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:33:32,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:33:32,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:33:32,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24247.60 MB 2025-02-14 19:33:32,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32686.62 MB 2025-02-14 19:33:32,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:33:32,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38166.07 MB 2025-02-14 19:33:32,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46556.77 MB 2025-02-14 19:33:32,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:33:32,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32686.62 MB 2025-02-14 19:33:33,058 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:33:33,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:33:33,060 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:33:33,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:33:33,061 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:33:33,065 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:33:33,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:33:33,066 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:33:33,066 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:35:02,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:02,005 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:35:02,012 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:35:02,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:02,019 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1987, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:35:02,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:02,021 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1987, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:35:32,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:35:32,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:35:32,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.72 seconds 2025-02-14 19:35:32,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:32,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26814.44 MB 2025-02-14 19:35:32,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33846.32 MB 2025-02-14 19:35:32,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7031.88 MB 2025-02-14 19:35:32,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59141.78 MB 2025-02-14 19:35:32,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37708.89 MB 2025-02-14 19:35:32,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21432.89 MB 2025-02-14 19:35:32,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42854.90 MB 2025-02-14 19:35:32,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:35:32,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:35:32,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:35:32,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:32,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33846.32 MB 2025-02-14 19:35:32,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26108.69 MB 2025-02-14 19:35:32,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7737.63 MB 2025-02-14 19:35:32,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37708.89 MB 2025-02-14 19:35:32,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63560.48 MB 2025-02-14 19:35:32,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25851.59 MB 2025-02-14 19:35:32,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53212.37 MB 2025-02-14 19:35:34,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:35:34,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:35:34,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 19:35:34,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:34,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26108.69 MB 2025-02-14 19:35:34,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26639.53 MB 2025-02-14 19:35:34,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:35:34,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63560.48 MB 2025-02-14 19:35:34,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32799.46 MB 2025-02-14 19:35:34,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30761.03 MB 2025-02-14 19:35:34,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30618.08 MB 2025-02-14 19:35:34,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:35:34,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:35:34,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:35:34,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:34,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26639.53 MB 2025-02-14 19:35:34,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28529.07 MB 2025-02-14 19:35:34,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:35:34,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32799.46 MB 2025-02-14 19:35:34,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32799.46 MB 2025-02-14 19:35:34,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:35:34,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29946.50 MB 2025-02-14 19:35:35,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:35:35,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:35:35,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:35:35,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:35,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28529.07 MB 2025-02-14 19:35:35,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30770.92 MB 2025-02-14 19:35:35,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:35:35,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32799.46 MB 2025-02-14 19:35:35,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38461.77 MB 2025-02-14 19:35:35,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:35:35,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36315.21 MB 2025-02-14 19:35:35,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:35:35,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:35:35,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:35:35,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:35,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26639.53 MB 2025-02-14 19:35:35,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30770.92 MB 2025-02-14 19:35:35,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:35:35,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32799.46 MB 2025-02-14 19:35:35,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38461.77 MB 2025-02-14 19:35:35,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:35:35,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36315.21 MB 2025-02-14 19:35:35,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:35:35,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:35:35,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:35:35,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:35,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32304.47 MB 2025-02-14 19:35:35,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33071.47 MB 2025-02-14 19:35:35,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:35:35,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38461.77 MB 2025-02-14 19:35:35,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:35:35,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 19:35:35,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33779.26 MB 2025-02-14 19:35:35,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:35:35,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:35:35,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:35:35,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:35,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33484.36 MB 2025-02-14 19:35:35,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33713.42 MB 2025-02-14 19:35:35,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-14 19:35:35,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38874.91 MB 2025-02-14 19:35:35,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:35:35,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:35:35,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33916.75 MB 2025-02-14 19:35:35,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:35:35,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:35:35,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.19 seconds 2025-02-14 19:35:35,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:35,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19891.57 MB 2025-02-14 19:35:35,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33913.90 MB 2025-02-14 19:35:35,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14022.33 MB 2025-02-14 19:35:35,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59141.78 MB 2025-02-14 19:35:35,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:35:35,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20266.88 MB 2025-02-14 19:35:35,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33916.75 MB 2025-02-14 19:35:35,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:35:35,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:35:35,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:35:35,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:35,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33913.90 MB 2025-02-14 19:35:35,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24885.75 MB 2025-02-14 19:35:35,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9028.15 MB 2025-02-14 19:35:35,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38874.91 MB 2025-02-14 19:35:35,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:35:35,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:35:35,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36417.27 MB 2025-02-14 19:35:35,499 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 19:35:35,500 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:35:35,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:35:35,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:35:35,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:35:35,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:35:35,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24885.75 MB 2025-02-14 19:35:35,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33296.57 MB 2025-02-14 19:35:35,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 19:35:35,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38874.91 MB 2025-02-14 19:35:35,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43056.63 MB 2025-02-14 19:35:35,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 19:35:35,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33296.57 MB 2025-02-14 19:35:35,661 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 19:35:35,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:35,662 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:35:35,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:35,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:35:35,668 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:35:35,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:35,669 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:35:35,669 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:35:44,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:44,557 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:35:44,562 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:35:44,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:44,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1960, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:35:44,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:35:44,566 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1960, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:36:15,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:36:15,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:36:15,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.59 seconds 2025-02-14 19:36:15,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:15,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26626.30 MB 2025-02-14 19:36:15,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33563.68 MB 2025-02-14 19:36:15,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6937.38 MB 2025-02-14 19:36:15,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51420.07 MB 2025-02-14 19:36:15,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37601.94 MB 2025-02-14 19:36:15,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13818.13 MB 2025-02-14 19:36:15,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42440.27 MB 2025-02-14 19:36:15,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:36:15,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:36:15,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 19:36:15,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:15,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33563.68 MB 2025-02-14 19:36:15,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25967.28 MB 2025-02-14 19:36:15,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7596.40 MB 2025-02-14 19:36:15,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37601.94 MB 2025-02-14 19:36:15,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63438.85 MB 2025-02-14 19:36:15,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25836.91 MB 2025-02-14 19:36:15,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53620.27 MB 2025-02-14 19:36:17,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:36:17,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:36:17,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 19:36:17,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25967.28 MB 2025-02-14 19:36:17,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26498.12 MB 2025-02-14 19:36:17,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:36:17,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63438.85 MB 2025-02-14 19:36:17,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 19:36:17,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31358.71 MB 2025-02-14 19:36:17,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30476.67 MB 2025-02-14 19:36:17,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:36:17,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:36:17,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:36:17,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 19:36:17,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28387.65 MB 2025-02-14 19:36:17,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:36:17,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 19:36:17,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 19:36:17,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:17,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29805.08 MB 2025-02-14 19:36:17,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:36:17,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:36:17,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:36:17,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28387.65 MB 2025-02-14 19:36:17,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 19:36:17,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:36:17,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 19:36:17,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38214.30 MB 2025-02-14 19:36:17,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:36:17,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 19:36:17,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:36:17,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:36:17,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:36:17,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 19:36:17,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 19:36:17,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:36:17,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 19:36:17,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38214.30 MB 2025-02-14 19:36:17,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:36:17,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 19:36:17,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:36:17,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:36:17,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:36:17,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32163.05 MB 2025-02-14 19:36:17,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32930.05 MB 2025-02-14 19:36:17,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:36:17,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38214.30 MB 2025-02-14 19:36:17,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38625.35 MB 2025-02-14 19:36:17,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 19:36:17,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33637.84 MB 2025-02-14 19:36:17,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:36:17,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:36:17,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:36:17,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33342.94 MB 2025-02-14 19:36:17,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33571.31 MB 2025-02-14 19:36:17,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-14 19:36:17,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38625.35 MB 2025-02-14 19:36:17,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38625.35 MB 2025-02-14 19:36:17,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:17,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.30 MB 2025-02-14 19:36:17,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:36:17,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:36:17,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.12 seconds 2025-02-14 19:36:17,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19797.50 MB 2025-02-14 19:36:17,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33771.85 MB 2025-02-14 19:36:17,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13974.34 MB 2025-02-14 19:36:17,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51420.07 MB 2025-02-14 19:36:17,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38625.35 MB 2025-02-14 19:36:17,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12794.72 MB 2025-02-14 19:36:17,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.30 MB 2025-02-14 19:36:17,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:36:17,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:36:17,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:36:17,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33771.85 MB 2025-02-14 19:36:17,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24793.51 MB 2025-02-14 19:36:17,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8978.33 MB 2025-02-14 19:36:17,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38625.35 MB 2025-02-14 19:36:17,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38625.35 MB 2025-02-14 19:36:17,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:17,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36276.76 MB 2025-02-14 19:36:17,980 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 19:36:17,980 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:36:17,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:36:17,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:36:17,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:36:17,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:17,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24793.51 MB 2025-02-14 19:36:17,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33210.11 MB 2025-02-14 19:36:17,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 19:36:17,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38625.35 MB 2025-02-14 19:36:17,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46992.98 MB 2025-02-14 19:36:17,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 19:36:17,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33210.11 MB 2025-02-14 19:36:18,143 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 19:36:18,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:18,144 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:36:18,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:18,145 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:36:18,150 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:36:18,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:18,151 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:36:18,151 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:36:27,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:27,500 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:36:27,505 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:36:27,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:27,508 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:36:27,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:27,509 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:36:29,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:36:29,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:36:29,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.41 seconds 2025-02-14 19:36:29,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:29,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-14 19:36:29,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-14 19:36:29,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 19:36:29,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55360.62 MB 2025-02-14 19:36:29,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23714.59 MB 2025-02-14 19:36:29,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31646.02 MB 2025-02-14 19:36:29,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23506.21 MB 2025-02-14 19:36:29,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:36:29,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:36:29,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:36:29,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:29,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-14 19:36:29,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14803.51 MB 2025-02-14 19:36:29,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.22 MB 2025-02-14 19:36:29,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 19:36:29,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23714.59 MB 2025-02-14 19:36:29,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:29,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16687.02 MB 2025-02-14 19:36:30,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:36:30,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:36:30,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 19:36:30,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:30,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14803.51 MB 2025-02-14 19:36:30,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14999.93 MB 2025-02-14 19:36:30,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-14 19:36:30,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 19:36:30,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23714.59 MB 2025-02-14 19:36:30,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:30,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18973.16 MB 2025-02-14 19:36:30,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:36:30,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:36:30,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:36:30,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:30,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14999.86 MB 2025-02-14 19:36:30,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15698.82 MB 2025-02-14 19:36:30,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-14 19:36:30,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 19:36:30,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23714.59 MB 2025-02-14 19:36:30,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:30,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16223.27 MB 2025-02-14 19:36:30,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:36:30,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:36:30,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:36:30,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:30,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15698.82 MB 2025-02-14 19:36:30,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16528.35 MB 2025-02-14 19:36:30,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 829.53 MB 2025-02-14 19:36:30,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 19:36:30,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23714.59 MB 2025-02-14 19:36:30,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:30,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18579.69 MB 2025-02-14 19:36:30,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:36:30,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:36:30,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:36:30,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:30,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14999.86 MB 2025-02-14 19:36:30,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16528.35 MB 2025-02-14 19:36:30,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1528.49 MB 2025-02-14 19:36:30,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 19:36:30,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23714.59 MB 2025-02-14 19:36:30,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:30,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18579.69 MB 2025-02-14 19:36:30,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:36:30,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:36:30,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:36:30,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:30,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17095.76 MB 2025-02-14 19:36:30,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17379.55 MB 2025-02-14 19:36:30,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.79 MB 2025-02-14 19:36:30,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 19:36:30,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23863.49 MB 2025-02-14 19:36:30,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 19:36:30,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17651.07 MB 2025-02-14 19:36:30,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:36:30,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:36:30,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:36:30,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:30,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17532.32 MB 2025-02-14 19:36:30,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17746.26 MB 2025-02-14 19:36:30,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.93 MB 2025-02-14 19:36:30,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23863.49 MB 2025-02-14 19:36:30,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23863.49 MB 2025-02-14 19:36:30,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:30,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17748.91 MB 2025-02-14 19:36:30,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:36:30,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:36:30,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.30 seconds 2025-02-14 19:36:30,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:30,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-14 19:36:30,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17947.23 MB 2025-02-14 19:36:30,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4445.46 MB 2025-02-14 19:36:30,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55360.62 MB 2025-02-14 19:36:30,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23863.49 MB 2025-02-14 19:36:30,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31497.13 MB 2025-02-14 19:36:30,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17947.23 MB 2025-02-14 19:36:31,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:36:31,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:36:31,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:36:31,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:31,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17947.23 MB 2025-02-14 19:36:31,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17315.37 MB 2025-02-14 19:36:31,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -631.86 MB 2025-02-14 19:36:31,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23863.49 MB 2025-02-14 19:36:31,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23863.49 MB 2025-02-14 19:36:31,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:31,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19051.82 MB 2025-02-14 19:36:31,103 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 19:36:31,104 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2,'] 2025-02-14 19:36:31,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:36:31,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:36:31,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:36:31,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:31,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17315.37 MB 2025-02-14 19:36:31,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25750.22 MB 2025-02-14 19:36:31,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 19:36:31,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23863.49 MB 2025-02-14 19:36:31,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32250.00 MB 2025-02-14 19:36:31,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 19:36:31,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25750.22 MB 2025-02-14 19:36:31,308 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 19:36:31,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:31,310 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:36:31,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:31,311 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:36:31,315 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:36:31,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:31,316 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:36:31,316 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2,'] 2025-02-14 19:36:49,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:49,875 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:36:49,880 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:36:49,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:49,883 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 165, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:36:49,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:49,884 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 165, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:36:52,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:36:52,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:36:52,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.56 seconds 2025-02-14 19:36:52,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:52,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14118.45 MB 2025-02-14 19:36:52,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14702.38 MB 2025-02-14 19:36:52,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 583.93 MB 2025-02-14 19:36:52,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44828.72 MB 2025-02-14 19:36:52,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19526.58 MB 2025-02-14 19:36:52,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25302.14 MB 2025-02-14 19:36:52,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23589.82 MB 2025-02-14 19:36:52,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:36:52,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:36:52,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:36:52,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:52,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.38 MB 2025-02-14 19:36:52,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14957.20 MB 2025-02-14 19:36:52,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 254.82 MB 2025-02-14 19:36:52,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19526.58 MB 2025-02-14 19:36:52,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19526.58 MB 2025-02-14 19:36:52,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:52,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16982.34 MB 2025-02-14 19:36:53,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:36:53,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:36:53,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 19:36:53,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14957.20 MB 2025-02-14 19:36:53,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15170.86 MB 2025-02-14 19:36:53,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 19:36:53,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19526.58 MB 2025-02-14 19:36:53,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19054.72 MB 2025-02-14 19:36:53,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 19:36:53,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19127.89 MB 2025-02-14 19:36:53,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:36:53,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:36:53,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:36:53,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15170.80 MB 2025-02-14 19:36:53,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15931.15 MB 2025-02-14 19:36:53,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 19:36:53,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19054.72 MB 2025-02-14 19:36:53,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19054.72 MB 2025-02-14 19:36:53,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:53,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16501.67 MB 2025-02-14 19:36:53,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:36:53,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:36:53,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:36:53,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15931.15 MB 2025-02-14 19:36:53,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16833.53 MB 2025-02-14 19:36:53,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 19:36:53,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19054.72 MB 2025-02-14 19:36:53,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20008.93 MB 2025-02-14 19:36:53,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 954.20 MB 2025-02-14 19:36:53,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19065.99 MB 2025-02-14 19:36:53,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:36:53,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:36:53,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:36:53,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15170.80 MB 2025-02-14 19:36:53,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16833.53 MB 2025-02-14 19:36:53,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 19:36:53,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19054.72 MB 2025-02-14 19:36:53,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20008.93 MB 2025-02-14 19:36:53,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 954.20 MB 2025-02-14 19:36:53,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19065.99 MB 2025-02-14 19:36:53,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:36:53,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:36:53,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:36:53,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17450.79 MB 2025-02-14 19:36:53,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17760.42 MB 2025-02-14 19:36:53,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-14 19:36:53,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20008.93 MB 2025-02-14 19:36:53,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20176.70 MB 2025-02-14 19:36:53,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 19:36:53,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18051.49 MB 2025-02-14 19:36:53,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:36:53,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:36:53,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:36:53,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17926.62 MB 2025-02-14 19:36:53,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18151.24 MB 2025-02-14 19:36:53,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.63 MB 2025-02-14 19:36:53,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20176.70 MB 2025-02-14 19:36:53,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20176.70 MB 2025-02-14 19:36:53,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:53,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18166.17 MB 2025-02-14 19:36:53,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:36:53,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:36:53,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.52 seconds 2025-02-14 19:36:53,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13543.58 MB 2025-02-14 19:36:53,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18351.95 MB 2025-02-14 19:36:53,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4808.37 MB 2025-02-14 19:36:53,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44828.72 MB 2025-02-14 19:36:53,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20176.70 MB 2025-02-14 19:36:53,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24652.02 MB 2025-02-14 19:36:53,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18351.95 MB 2025-02-14 19:36:53,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:36:53,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:36:53,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:36:53,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18351.95 MB 2025-02-14 19:36:53,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17415.26 MB 2025-02-14 19:36:53,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -936.68 MB 2025-02-14 19:36:53,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20176.70 MB 2025-02-14 19:36:53,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20176.70 MB 2025-02-14 19:36:53,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:36:53,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19154.20 MB 2025-02-14 19:36:53,692 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 19:36:53,693 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:36:53,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:36:53,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:36:53,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:36:53,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:36:53,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17415.26 MB 2025-02-14 19:36:53,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25838.47 MB 2025-02-14 19:36:53,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 19:36:53,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20176.70 MB 2025-02-14 19:36:53,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30647.78 MB 2025-02-14 19:36:53,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 19:36:53,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25838.47 MB 2025-02-14 19:36:53,854 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 19:36:53,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:53,856 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:36:53,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:53,857 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:36:53,861 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:36:53,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:36:53,862 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:36:53,862 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:38:26,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:38:26,556 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:38:26,561 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:38:26,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:38:26,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 354, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:38:26,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:38:26,566 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 354, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:38:31,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:38:31,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:38:31,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.41 seconds 2025-02-14 19:38:31,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:31,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15435.44 MB 2025-02-14 19:38:31,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16688.22 MB 2025-02-14 19:38:31,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1252.79 MB 2025-02-14 19:38:31,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39023.80 MB 2025-02-14 19:38:31,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19054.72 MB 2025-02-14 19:38:31,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19969.08 MB 2025-02-14 19:38:31,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25586.28 MB 2025-02-14 19:38:32,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:38:32,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:38:32,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:38:32,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:32,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16688.22 MB 2025-02-14 19:38:32,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17289.58 MB 2025-02-14 19:38:32,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 601.36 MB 2025-02-14 19:38:32,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19054.72 MB 2025-02-14 19:38:32,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23823.65 MB 2025-02-14 19:38:32,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4768.92 MB 2025-02-14 19:38:32,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21656.83 MB 2025-02-14 19:38:33,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:38:33,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:38:33,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.67 seconds 2025-02-14 19:38:33,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:33,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17289.58 MB 2025-02-14 19:38:33,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17758.05 MB 2025-02-14 19:38:33,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 468.47 MB 2025-02-14 19:38:33,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23823.65 MB 2025-02-14 19:38:33,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20910.70 MB 2025-02-14 19:38:33,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2912.94 MB 2025-02-14 19:38:33,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21715.07 MB 2025-02-14 19:38:33,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:38:33,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:38:33,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:38:33,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:33,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17758.05 MB 2025-02-14 19:38:33,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19425.55 MB 2025-02-14 19:38:33,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1667.50 MB 2025-02-14 19:38:33,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20910.70 MB 2025-02-14 19:38:33,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22580.04 MB 2025-02-14 19:38:33,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1669.33 MB 2025-02-14 19:38:33,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20676.43 MB 2025-02-14 19:38:33,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:38:33,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:38:33,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 19:38:33,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:33,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19425.55 MB 2025-02-14 19:38:33,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21403.99 MB 2025-02-14 19:38:33,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1978.45 MB 2025-02-14 19:38:33,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22580.04 MB 2025-02-14 19:38:33,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28007.46 MB 2025-02-14 19:38:33,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5427.43 MB 2025-02-14 19:38:33,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26297.73 MB 2025-02-14 19:38:33,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:38:33,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:38:33,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:38:33,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:33,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17758.05 MB 2025-02-14 19:38:33,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21403.99 MB 2025-02-14 19:38:33,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3645.94 MB 2025-02-14 19:38:33,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20910.70 MB 2025-02-14 19:38:33,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28007.46 MB 2025-02-14 19:38:33,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7096.76 MB 2025-02-14 19:38:33,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26297.73 MB 2025-02-14 19:38:34,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:38:34,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:38:34,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 19:38:34,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:34,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22757.34 MB 2025-02-14 19:38:34,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23435.14 MB 2025-02-14 19:38:34,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 677.80 MB 2025-02-14 19:38:34,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28007.46 MB 2025-02-14 19:38:34,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28374.47 MB 2025-02-14 19:38:34,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 367.00 MB 2025-02-14 19:38:34,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24059.76 MB 2025-02-14 19:38:34,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:38:34,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:38:34,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:38:34,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:34,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23799.52 MB 2025-02-14 19:38:34,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24031.07 MB 2025-02-14 19:38:34,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.56 MB 2025-02-14 19:38:34,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28374.47 MB 2025-02-14 19:38:34,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28374.47 MB 2025-02-14 19:38:34,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:38:34,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24192.55 MB 2025-02-14 19:38:34,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:38:34,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:38:34,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.48 seconds 2025-02-14 19:38:34,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:34,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14202.07 MB 2025-02-14 19:38:34,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24232.14 MB 2025-02-14 19:38:34,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10030.07 MB 2025-02-14 19:38:34,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39023.80 MB 2025-02-14 19:38:34,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28374.47 MB 2025-02-14 19:38:34,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10649.34 MB 2025-02-14 19:38:34,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24232.14 MB 2025-02-14 19:38:34,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:38:34,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:38:34,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 19:38:34,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:34,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24232.14 MB 2025-02-14 19:38:34,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18985.31 MB 2025-02-14 19:38:34,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5246.84 MB 2025-02-14 19:38:34,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28374.47 MB 2025-02-14 19:38:34,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28374.47 MB 2025-02-14 19:38:34,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:38:34,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27447.08 MB 2025-02-14 19:38:34,328 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:38:34,328 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1,'] 2025-02-14 19:38:34,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:38:34,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:38:34,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:38:34,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:38:34,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18985.31 MB 2025-02-14 19:38:34,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27424.33 MB 2025-02-14 19:38:34,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:38:34,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28374.47 MB 2025-02-14 19:38:34,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38864.42 MB 2025-02-14 19:38:34,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 19:38:34,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27424.33 MB 2025-02-14 19:38:34,499 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:38:34,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:38:34,501 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:38:34,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:38:34,502 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:38:34,508 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:38:34,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:38:34,509 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:38:34,509 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1,'] 2025-02-14 19:39:53,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:39:53,977 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:39:53,983 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:39:53,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:39:53,988 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2390, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:39:53,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:39:53,990 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2390, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:40:30,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:40:30,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:40:30,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.75 seconds 2025-02-14 19:40:30,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:30,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29624.44 MB 2025-02-14 19:40:30,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38082.52 MB 2025-02-14 19:40:30,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8458.08 MB 2025-02-14 19:40:30,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59909.34 MB 2025-02-14 19:40:30,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43421.53 MB 2025-02-14 19:40:30,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16487.81 MB 2025-02-14 19:40:30,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47023.05 MB 2025-02-14 19:40:30,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:40:30,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:40:30,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 19:40:30,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:30,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38082.52 MB 2025-02-14 19:40:30,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28203.63 MB 2025-02-14 19:40:30,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9878.89 MB 2025-02-14 19:40:30,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43421.53 MB 2025-02-14 19:40:30,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70749.52 MB 2025-02-14 19:40:30,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 27327.99 MB 2025-02-14 19:40:30,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61313.00 MB 2025-02-14 19:40:32,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:40:32,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:40:32,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 19:40:32,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:32,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28203.63 MB 2025-02-14 19:40:32,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28734.47 MB 2025-02-14 19:40:32,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:40:32,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70749.52 MB 2025-02-14 19:40:32,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32147.24 MB 2025-02-14 19:40:32,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38602.28 MB 2025-02-14 19:40:32,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32714.06 MB 2025-02-14 19:40:32,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:40:32,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:40:32,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:40:32,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:32,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28734.47 MB 2025-02-14 19:40:32,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30624.01 MB 2025-02-14 19:40:32,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:40:32,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32147.24 MB 2025-02-14 19:40:32,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34034.68 MB 2025-02-14 19:40:32,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:40:32,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32041.44 MB 2025-02-14 19:40:33,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:40:33,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:40:33,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:40:33,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:33,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30624.01 MB 2025-02-14 19:40:33,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32865.86 MB 2025-02-14 19:40:33,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:40:33,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34034.68 MB 2025-02-14 19:40:33,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40168.85 MB 2025-02-14 19:40:33,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:40:33,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38410.15 MB 2025-02-14 19:40:33,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:40:33,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:40:33,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:40:33,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:33,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28734.47 MB 2025-02-14 19:40:33,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32865.86 MB 2025-02-14 19:40:33,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:40:33,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32147.24 MB 2025-02-14 19:40:33,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40168.85 MB 2025-02-14 19:40:33,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 19:40:33,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38410.15 MB 2025-02-14 19:40:33,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:40:33,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:40:33,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:40:33,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:33,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34399.41 MB 2025-02-14 19:40:33,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35166.41 MB 2025-02-14 19:40:33,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:40:33,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40168.85 MB 2025-02-14 19:40:33,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40586.18 MB 2025-02-14 19:40:33,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:40:33,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35874.20 MB 2025-02-14 19:40:33,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:40:33,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:40:33,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:40:33,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:33,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35579.30 MB 2025-02-14 19:40:33,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35808.04 MB 2025-02-14 19:40:33,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-14 19:40:33,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40586.18 MB 2025-02-14 19:40:33,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40586.18 MB 2025-02-14 19:40:33,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:40:33,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36013.02 MB 2025-02-14 19:40:33,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:40:33,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:40:33,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.26 seconds 2025-02-14 19:40:33,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:33,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21296.57 MB 2025-02-14 19:40:33,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36008.69 MB 2025-02-14 19:40:33,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14712.12 MB 2025-02-14 19:40:33,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55679.39 MB 2025-02-14 19:40:33,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40586.18 MB 2025-02-14 19:40:33,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15093.20 MB 2025-02-14 19:40:33,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36013.02 MB 2025-02-14 19:40:33,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:40:33,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:40:33,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:40:33,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:33,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36008.69 MB 2025-02-14 19:40:33,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26294.49 MB 2025-02-14 19:40:33,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9714.20 MB 2025-02-14 19:40:33,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40586.18 MB 2025-02-14 19:40:33,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40586.18 MB 2025-02-14 19:40:33,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:40:33,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38515.14 MB 2025-02-14 19:40:33,541 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 19:40:33,541 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 19:40:33,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:40:33,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:40:33,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:40:33,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:40:33,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26294.49 MB 2025-02-14 19:40:33,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34716.30 MB 2025-02-14 19:40:33,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.81 MB 2025-02-14 19:40:33,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40586.18 MB 2025-02-14 19:40:33,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44772.10 MB 2025-02-14 19:40:33,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-14 19:40:33,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34716.30 MB 2025-02-14 19:40:33,703 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 19:40:33,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:40:33,705 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:40:33,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:40:33,706 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:40:33,710 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:40:33,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:40:33,711 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:40:33,711 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 19:40:59,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:40:59,815 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:40:59,819 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:40:59,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:40:59,823 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1986, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:40:59,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:40:59,824 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1986, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:41:30,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:41:30,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:41:30,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.95 seconds 2025-02-14 19:41:30,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:30,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26807.47 MB 2025-02-14 19:41:30,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33835.81 MB 2025-02-14 19:41:30,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7028.34 MB 2025-02-14 19:41:30,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53143.93 MB 2025-02-14 19:41:30,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37702.60 MB 2025-02-14 19:41:30,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15441.33 MB 2025-02-14 19:41:30,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42847.93 MB 2025-02-14 19:41:30,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:41:30,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:41:30,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:41:30,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:30,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33835.81 MB 2025-02-14 19:41:30,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26103.49 MB 2025-02-14 19:41:30,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7732.32 MB 2025-02-14 19:41:30,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37702.60 MB 2025-02-14 19:41:30,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64040.73 MB 2025-02-14 19:41:30,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 26338.13 MB 2025-02-14 19:41:30,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53566.28 MB 2025-02-14 19:41:32,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:41:32,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:41:32,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 19:41:32,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:32,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26103.49 MB 2025-02-14 19:41:32,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26634.34 MB 2025-02-14 19:41:32,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:41:32,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64040.73 MB 2025-02-14 19:41:32,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32797.36 MB 2025-02-14 19:41:32,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31243.37 MB 2025-02-14 19:41:32,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30612.88 MB 2025-02-14 19:41:32,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:41:32,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:41:32,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:41:32,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:32,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26634.34 MB 2025-02-14 19:41:32,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28523.87 MB 2025-02-14 19:41:32,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:41:32,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32797.36 MB 2025-02-14 19:41:32,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32797.36 MB 2025-02-14 19:41:32,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:41:32,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29941.30 MB 2025-02-14 19:41:33,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:41:33,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:41:33,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:41:33,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:33,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28523.87 MB 2025-02-14 19:41:33,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30765.73 MB 2025-02-14 19:41:33,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:41:33,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32797.36 MB 2025-02-14 19:41:33,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38459.67 MB 2025-02-14 19:41:33,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:41:33,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36310.01 MB 2025-02-14 19:41:33,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:41:33,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:41:33,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:41:33,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:33,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26634.34 MB 2025-02-14 19:41:33,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30765.73 MB 2025-02-14 19:41:33,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:41:33,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32797.36 MB 2025-02-14 19:41:33,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38459.67 MB 2025-02-14 19:41:33,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:41:33,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36310.01 MB 2025-02-14 19:41:33,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:41:33,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:41:33,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:41:33,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:33,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32299.27 MB 2025-02-14 19:41:33,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33066.27 MB 2025-02-14 19:41:33,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:41:33,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38459.67 MB 2025-02-14 19:41:33,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:41:33,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:41:33,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33774.06 MB 2025-02-14 19:41:33,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:41:33,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:41:33,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:41:33,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:33,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33479.16 MB 2025-02-14 19:41:33,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33707.86 MB 2025-02-14 19:41:33,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.70 MB 2025-02-14 19:41:33,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38874.91 MB 2025-02-14 19:41:33,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:41:33,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:41:33,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33913.78 MB 2025-02-14 19:41:33,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:41:33,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:41:33,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.43 seconds 2025-02-14 19:41:33,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:33,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19888.09 MB 2025-02-14 19:41:33,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33908.02 MB 2025-02-14 19:41:33,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14019.93 MB 2025-02-14 19:41:33,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53143.93 MB 2025-02-14 19:41:33,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:41:33,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14269.02 MB 2025-02-14 19:41:33,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33913.78 MB 2025-02-14 19:41:33,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:41:33,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:41:33,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:41:33,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:33,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33908.02 MB 2025-02-14 19:41:33,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.70 MB 2025-02-14 19:41:33,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9029.32 MB 2025-02-14 19:41:33,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38874.91 MB 2025-02-14 19:41:33,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38874.91 MB 2025-02-14 19:41:33,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:41:33,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36408.64 MB 2025-02-14 19:41:33,540 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 19:41:33,540 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:41:33,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:41:33,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:41:33,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:41:33,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:41:33,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.70 MB 2025-02-14 19:41:33,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33279.64 MB 2025-02-14 19:41:33,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 19:41:33,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38874.91 MB 2025-02-14 19:41:33,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47227.86 MB 2025-02-14 19:41:33,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 19:41:33,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33279.64 MB 2025-02-14 19:41:33,701 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 19:41:33,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:41:33,703 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:41:33,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:41:33,704 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:41:33,708 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:41:33,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:41:33,709 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:41:33,710 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:42:38,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:42:38,656 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:42:38,665 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:42:38,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:42:38,672 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 409, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:42:38,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:42:38,674 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 409, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:42:45,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:42:45,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:42:45,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.41 seconds 2025-02-14 19:42:45,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:45,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15818.68 MB 2025-02-14 19:42:45,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17266.11 MB 2025-02-14 19:42:45,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1447.43 MB 2025-02-14 19:42:45,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59756.25 MB 2025-02-14 19:42:45,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 19:42:45,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31585.21 MB 2025-02-14 19:42:45,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26196.02 MB 2025-02-14 19:42:45,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:42:45,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:42:45,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:42:45,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:45,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17266.11 MB 2025-02-14 19:42:45,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17805.79 MB 2025-02-14 19:42:45,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 539.68 MB 2025-02-14 19:42:45,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 19:42:45,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28171.04 MB 2025-02-14 19:42:45,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:42:45,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22719.74 MB 2025-02-14 19:42:46,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:42:46,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:42:46,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.85 seconds 2025-02-14 19:42:46,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:46,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17805.79 MB 2025-02-14 19:42:46,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18318.05 MB 2025-02-14 19:42:46,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 512.26 MB 2025-02-14 19:42:46,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28171.04 MB 2025-02-14 19:42:46,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27227.32 MB 2025-02-14 19:42:46,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 19:42:46,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22315.55 MB 2025-02-14 19:42:46,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:42:46,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:42:46,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:42:46,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:46,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18318.05 MB 2025-02-14 19:42:46,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20141.53 MB 2025-02-14 19:42:46,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1823.47 MB 2025-02-14 19:42:46,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27227.32 MB 2025-02-14 19:42:46,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27227.32 MB 2025-02-14 19:42:46,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:42:46,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21509.35 MB 2025-02-14 19:42:47,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:42:47,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:42:47,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:42:47,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:47,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20141.53 MB 2025-02-14 19:42:47,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22304.92 MB 2025-02-14 19:42:47,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2163.39 MB 2025-02-14 19:42:47,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27227.32 MB 2025-02-14 19:42:47,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30421.29 MB 2025-02-14 19:42:47,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3193.96 MB 2025-02-14 19:42:47,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27655.15 MB 2025-02-14 19:42:47,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:42:47,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:42:47,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:42:47,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:47,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18318.05 MB 2025-02-14 19:42:47,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22304.92 MB 2025-02-14 19:42:47,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3986.87 MB 2025-02-14 19:42:47,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27227.32 MB 2025-02-14 19:42:47,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30421.29 MB 2025-02-14 19:42:47,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3193.96 MB 2025-02-14 19:42:47,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27655.15 MB 2025-02-14 19:42:47,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:42:47,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:42:47,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:42:47,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:47,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23784.79 MB 2025-02-14 19:42:47,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24524.95 MB 2025-02-14 19:42:47,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 740.16 MB 2025-02-14 19:42:47,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30421.29 MB 2025-02-14 19:42:47,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30823.94 MB 2025-02-14 19:42:47,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 19:42:47,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25207.96 MB 2025-02-14 19:42:47,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:42:47,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:42:47,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:42:47,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:47,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24923.39 MB 2025-02-14 19:42:47,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25130.29 MB 2025-02-14 19:42:47,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.90 MB 2025-02-14 19:42:47,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30823.94 MB 2025-02-14 19:42:47,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30828.13 MB 2025-02-14 19:42:47,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 19:42:47,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25302.84 MB 2025-02-14 19:42:47,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:42:47,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:42:47,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.68 seconds 2025-02-14 19:42:47,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:47,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14393.70 MB 2025-02-14 19:42:47,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25331.36 MB 2025-02-14 19:42:47,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10937.66 MB 2025-02-14 19:42:47,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59756.25 MB 2025-02-14 19:42:47,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30828.13 MB 2025-02-14 19:42:47,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28928.11 MB 2025-02-14 19:42:47,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25331.36 MB 2025-02-14 19:42:47,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:42:47,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:42:47,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:42:47,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:47,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25331.36 MB 2025-02-14 19:42:47,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19331.75 MB 2025-02-14 19:42:47,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5999.61 MB 2025-02-14 19:42:47,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30828.13 MB 2025-02-14 19:42:47,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30828.13 MB 2025-02-14 19:42:47,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:42:47,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28043.96 MB 2025-02-14 19:42:47,642 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:42:47,642 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:42:47,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:42:47,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:42:47,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:42:47,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:42:47,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19331.75 MB 2025-02-14 19:42:47,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27770.78 MB 2025-02-14 19:42:47,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:42:47,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30828.13 MB 2025-02-14 19:42:47,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39218.84 MB 2025-02-14 19:42:47,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:42:47,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27770.78 MB 2025-02-14 19:42:47,804 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:42:47,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:42:47,805 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:42:47,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:42:47,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:42:47,811 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:42:47,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:42:47,812 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:42:47,812 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:43:39,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:43:39,957 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:43:39,962 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:43:39,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:43:39,966 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1445, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:43:39,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:43:39,967 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1445, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:44:02,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:44:02,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:44:02,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.18 seconds 2025-02-14 19:44:02,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:02,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23037.70 MB 2025-02-14 19:44:02,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28151.47 MB 2025-02-14 19:44:02,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5113.77 MB 2025-02-14 19:44:02,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51803.85 MB 2025-02-14 19:44:02,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34886.12 MB 2025-02-14 19:44:02,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16917.73 MB 2025-02-14 19:44:02,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37038.92 MB 2025-02-14 19:44:02,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:44:02,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:44:02,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:44:02,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:02,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28151.47 MB 2025-02-14 19:44:02,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23289.96 MB 2025-02-14 19:44:02,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4861.52 MB 2025-02-14 19:44:02,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34886.12 MB 2025-02-14 19:44:02,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47624.22 MB 2025-02-14 19:44:02,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12738.10 MB 2025-02-14 19:44:02,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42640.38 MB 2025-02-14 19:44:04,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:44:04,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:44:04,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 19:44:04,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23289.96 MB 2025-02-14 19:44:04,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23820.80 MB 2025-02-14 19:44:04,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:44:04,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47624.22 MB 2025-02-14 19:44:04,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25585.25 MB 2025-02-14 19:44:04,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22038.97 MB 2025-02-14 19:44:04,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27800.38 MB 2025-02-14 19:44:04,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:44:04,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:44:04,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:44:04,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23820.80 MB 2025-02-14 19:44:04,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25710.33 MB 2025-02-14 19:44:04,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:44:04,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25585.25 MB 2025-02-14 19:44:04,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28888.27 MB 2025-02-14 19:44:04,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 19:44:04,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27127.76 MB 2025-02-14 19:44:04,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:44:04,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:44:04,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:44:04,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25710.33 MB 2025-02-14 19:44:04,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27952.19 MB 2025-02-14 19:44:04,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:44:04,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28888.27 MB 2025-02-14 19:44:04,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35022.44 MB 2025-02-14 19:44:04,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:44:04,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.47 MB 2025-02-14 19:44:04,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:44:04,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:44:04,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:44:04,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23820.80 MB 2025-02-14 19:44:04,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27952.19 MB 2025-02-14 19:44:04,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:44:04,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25585.25 MB 2025-02-14 19:44:04,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35022.44 MB 2025-02-14 19:44:04,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 19:44:04,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.47 MB 2025-02-14 19:44:04,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:44:04,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:44:04,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:44:04,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29485.73 MB 2025-02-14 19:44:04,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30252.73 MB 2025-02-14 19:44:04,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:44:04,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35022.44 MB 2025-02-14 19:44:04,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35437.67 MB 2025-02-14 19:44:04,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:44:04,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30960.52 MB 2025-02-14 19:44:04,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:44:04,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:44:04,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:44:04,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30665.62 MB 2025-02-14 19:44:04,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30894.09 MB 2025-02-14 19:44:04,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 19:44:04,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35437.67 MB 2025-02-14 19:44:04,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35437.67 MB 2025-02-14 19:44:04,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:44:04,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31105.39 MB 2025-02-14 19:44:04,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:44:04,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:44:04,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.63 seconds 2025-02-14 19:44:04,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18003.20 MB 2025-02-14 19:44:04,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31095.16 MB 2025-02-14 19:44:04,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13091.96 MB 2025-02-14 19:44:04,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51803.85 MB 2025-02-14 19:44:04,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35437.67 MB 2025-02-14 19:44:04,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16366.17 MB 2025-02-14 19:44:04,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31105.39 MB 2025-02-14 19:44:04,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:44:04,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:44:04,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:44:04,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31095.16 MB 2025-02-14 19:44:04,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23007.59 MB 2025-02-14 19:44:04,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8087.57 MB 2025-02-14 19:44:04,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35437.67 MB 2025-02-14 19:44:04,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35437.67 MB 2025-02-14 19:44:04,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:44:04,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33606.83 MB 2025-02-14 19:44:04,883 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:44:04,883 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 19:44:04,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:44:04,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:44:04,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:44:04,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:44:04,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23007.59 MB 2025-02-14 19:44:04,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31446.61 MB 2025-02-14 19:44:04,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:44:04,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35437.67 MB 2025-02-14 19:44:04,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39634.08 MB 2025-02-14 19:44:04,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 19:44:04,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31446.61 MB 2025-02-14 19:44:05,050 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:44:05,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:44:05,052 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:44:05,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:44:05,053 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:44:05,057 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:44:05,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:44:05,058 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:44:05,058 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 19:45:21,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:45:21,321 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:45:21,326 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:45:21,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:45:21,332 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1239, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:45:21,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:45:21,333 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1239, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:45:40,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:45:40,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:45:40,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.02 seconds 2025-02-14 19:45:40,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:40,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21602.26 MB 2025-02-14 19:45:40,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25987.40 MB 2025-02-14 19:45:40,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4385.14 MB 2025-02-14 19:45:40,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52219.08 MB 2025-02-14 19:45:40,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34166.80 MB 2025-02-14 19:45:40,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18052.28 MB 2025-02-14 19:45:40,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34924.00 MB 2025-02-14 19:45:40,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:45:40,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:45:40,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 19:45:40,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:40,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25987.40 MB 2025-02-14 19:45:40,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21853.83 MB 2025-02-14 19:45:40,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4133.57 MB 2025-02-14 19:45:40,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34166.80 MB 2025-02-14 19:45:40,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34166.80 MB 2025-02-14 19:45:40,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:45:40,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30900.20 MB 2025-02-14 19:45:42,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:45:42,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:45:42,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.66 seconds 2025-02-14 19:45:42,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21853.83 MB 2025-02-14 19:45:42,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22315.66 MB 2025-02-14 19:45:42,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 461.83 MB 2025-02-14 19:45:42,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34166.80 MB 2025-02-14 19:45:42,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29781.66 MB 2025-02-14 19:45:42,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4385.14 MB 2025-02-14 19:45:42,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26278.28 MB 2025-02-14 19:45:42,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:45:42,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:45:42,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:45:42,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22315.66 MB 2025-02-14 19:45:42,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23959.83 MB 2025-02-14 19:45:42,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1644.17 MB 2025-02-14 19:45:42,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29781.66 MB 2025-02-14 19:45:42,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29781.66 MB 2025-02-14 19:45:42,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:45:42,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25192.99 MB 2025-02-14 19:45:42,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:45:42,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:45:42,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 19:45:42,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23959.83 MB 2025-02-14 19:45:42,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25910.25 MB 2025-02-14 19:45:42,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1950.42 MB 2025-02-14 19:45:42,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29781.66 MB 2025-02-14 19:45:42,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32658.95 MB 2025-02-14 19:45:42,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2877.29 MB 2025-02-14 19:45:42,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30737.44 MB 2025-02-14 19:45:42,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:45:42,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:45:42,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 19:45:42,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22315.66 MB 2025-02-14 19:45:42,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25910.25 MB 2025-02-14 19:45:42,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3594.59 MB 2025-02-14 19:45:42,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29781.66 MB 2025-02-14 19:45:42,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32658.95 MB 2025-02-14 19:45:42,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2877.29 MB 2025-02-14 19:45:42,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30737.44 MB 2025-02-14 19:45:42,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:45:42,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:45:42,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 19:45:42,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27244.43 MB 2025-02-14 19:45:42,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27913.30 MB 2025-02-14 19:45:42,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 668.86 MB 2025-02-14 19:45:42,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32658.95 MB 2025-02-14 19:45:42,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33017.56 MB 2025-02-14 19:45:42,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 358.61 MB 2025-02-14 19:45:42,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28529.07 MB 2025-02-14 19:45:42,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:45:42,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:45:42,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:45:42,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28272.51 MB 2025-02-14 19:45:42,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28490.93 MB 2025-02-14 19:45:42,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.42 MB 2025-02-14 19:45:42,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33017.56 MB 2025-02-14 19:45:42,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33017.56 MB 2025-02-14 19:45:42,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:45:42,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28616.21 MB 2025-02-14 19:45:42,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:45:42,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:45:42,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.11 seconds 2025-02-14 19:45:42,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17285.48 MB 2025-02-14 19:45:42,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28692.00 MB 2025-02-14 19:45:42,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11406.52 MB 2025-02-14 19:45:42,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52219.08 MB 2025-02-14 19:45:42,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33017.56 MB 2025-02-14 19:45:42,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19201.52 MB 2025-02-14 19:45:42,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28692.00 MB 2025-02-14 19:45:42,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:45:42,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:45:42,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:45:42,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28692.00 MB 2025-02-14 19:45:42,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31706.04 MB 2025-02-14 19:45:42,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 19:45:42,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33017.56 MB 2025-02-14 19:45:42,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33017.56 MB 2025-02-14 19:45:42,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:45:42,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32007.41 MB 2025-02-14 19:45:42,735 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:45:42,736 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:45:42,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:45:42,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:45:42,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:45:42,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:45:42,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22046.04 MB 2025-02-14 19:45:42,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30485.06 MB 2025-02-14 19:45:42,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:45:42,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33017.56 MB 2025-02-14 19:45:42,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41408.27 MB 2025-02-14 19:45:42,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:45:42,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30485.06 MB 2025-02-14 19:45:42,907 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:45:42,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:45:42,908 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:45:42,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:45:42,909 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:45:42,914 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:45:42,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:45:42,915 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:45:42,915 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:46:34,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:46:34,285 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:46:34,290 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:46:34,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:46:34,294 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1605, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:46:34,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:46:34,295 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1605, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:46:59,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:46:59,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:46:59,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.75 seconds 2025-02-14 19:46:59,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:46:59,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24152.60 MB 2025-02-14 19:46:59,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29832.61 MB 2025-02-14 19:46:59,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5680.01 MB 2025-02-14 19:46:59,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53993.28 MB 2025-02-14 19:46:59,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35462.84 MB 2025-02-14 19:46:59,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18530.44 MB 2025-02-14 19:46:59,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38834.11 MB 2025-02-14 19:46:59,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:46:59,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:46:59,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:46:59,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:46:59,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29832.61 MB 2025-02-14 19:46:59,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24121.75 MB 2025-02-14 19:46:59,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5710.86 MB 2025-02-14 19:46:59,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35462.84 MB 2025-02-14 19:46:59,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52909.05 MB 2025-02-14 19:46:59,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17446.21 MB 2025-02-14 19:46:59,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45908.38 MB 2025-02-14 19:47:01,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:47:01,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:47:01,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 19:47:01,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24121.75 MB 2025-02-14 19:47:01,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24652.59 MB 2025-02-14 19:47:01,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:47:01,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52909.05 MB 2025-02-14 19:47:01,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27921.48 MB 2025-02-14 19:47:01,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24987.57 MB 2025-02-14 19:47:01,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28632.17 MB 2025-02-14 19:47:01,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:47:01,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:47:01,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:47:01,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24652.59 MB 2025-02-14 19:47:01,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26542.12 MB 2025-02-14 19:47:01,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:47:01,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27921.48 MB 2025-02-14 19:47:01,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29808.92 MB 2025-02-14 19:47:01,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:47:01,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27959.55 MB 2025-02-14 19:47:01,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:47:01,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:47:01,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:47:01,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26542.12 MB 2025-02-14 19:47:01,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28783.98 MB 2025-02-14 19:47:01,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:47:01,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29808.92 MB 2025-02-14 19:47:01,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35943.09 MB 2025-02-14 19:47:01,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:47:01,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34328.26 MB 2025-02-14 19:47:01,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:47:01,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:47:01,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 19:47:01,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24652.59 MB 2025-02-14 19:47:01,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28783.98 MB 2025-02-14 19:47:01,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:47:01,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27921.48 MB 2025-02-14 19:47:01,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35943.09 MB 2025-02-14 19:47:01,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 19:47:01,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34328.26 MB 2025-02-14 19:47:01,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:47:01,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:47:01,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:47:01,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30317.52 MB 2025-02-14 19:47:01,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31084.52 MB 2025-02-14 19:47:01,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:47:01,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35943.09 MB 2025-02-14 19:47:01,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36356.23 MB 2025-02-14 19:47:01,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 19:47:01,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31792.31 MB 2025-02-14 19:47:01,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:47:01,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:47:01,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:47:01,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31497.41 MB 2025-02-14 19:47:01,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31724.45 MB 2025-02-14 19:47:01,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.04 MB 2025-02-14 19:47:01,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36356.23 MB 2025-02-14 19:47:01,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36356.23 MB 2025-02-14 19:47:01,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:47:01,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31954.64 MB 2025-02-14 19:47:01,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:47:01,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:47:01,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.28 seconds 2025-02-14 19:47:01,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18560.65 MB 2025-02-14 19:47:01,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31924.47 MB 2025-02-14 19:47:01,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13363.81 MB 2025-02-14 19:47:01,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53993.28 MB 2025-02-14 19:47:01,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36356.23 MB 2025-02-14 19:47:01,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17637.05 MB 2025-02-14 19:47:01,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31954.64 MB 2025-02-14 19:47:01,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:47:01,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:47:01,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:47:01,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31924.47 MB 2025-02-14 19:47:01,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23549.13 MB 2025-02-14 19:47:01,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8375.34 MB 2025-02-14 19:47:01,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36356.23 MB 2025-02-14 19:47:01,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36356.23 MB 2025-02-14 19:47:01,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:47:01,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34423.39 MB 2025-02-14 19:47:01,864 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 19:47:01,864 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1,'] 2025-02-14 19:47:01,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:47:01,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:47:01,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:47:01,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:47:01,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23549.13 MB 2025-02-14 19:47:01,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31944.35 MB 2025-02-14 19:47:01,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 19:47:01,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36356.23 MB 2025-02-14 19:47:01,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44702.89 MB 2025-02-14 19:47:01,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 19:47:01,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31944.35 MB 2025-02-14 19:47:02,035 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 19:47:02,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:47:02,036 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:47:02,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:47:02,037 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:47:02,042 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:47:02,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:47:02,043 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:47:02,043 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1,'] 2025-02-14 19:48:46,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:48:46,473 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:48:46,481 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:48:46,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:48:46,488 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:48:46,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:48:46,490 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:49:03,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:49:03,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:49:03,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.99 seconds 2025-02-14 19:49:03,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:03,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20682.46 MB 2025-02-14 19:49:03,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24600.07 MB 2025-02-14 19:49:03,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3917.61 MB 2025-02-14 19:49:03,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53049.56 MB 2025-02-14 19:49:03,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 19:49:03,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26566.72 MB 2025-02-14 19:49:03,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33551.22 MB 2025-02-14 19:49:03,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:49:03,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:49:03,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:49:03,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:03,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24600.07 MB 2025-02-14 19:49:03,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21533.85 MB 2025-02-14 19:49:03,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3066.22 MB 2025-02-14 19:49:03,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 19:49:03,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43408.95 MB 2025-02-14 19:49:03,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16926.11 MB 2025-02-14 19:49:03,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35614.85 MB 2025-02-14 19:49:05,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:49:05,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:49:05,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 19:49:05,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:05,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21533.85 MB 2025-02-14 19:49:05,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22064.69 MB 2025-02-14 19:49:05,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:49:05,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43408.95 MB 2025-02-14 19:49:05,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29316.09 MB 2025-02-14 19:49:05,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14092.86 MB 2025-02-14 19:49:05,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26043.24 MB 2025-02-14 19:49:05,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:49:05,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:49:05,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:49:05,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:05,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22064.69 MB 2025-02-14 19:49:05,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23954.22 MB 2025-02-14 19:49:05,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:49:05,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29316.09 MB 2025-02-14 19:49:05,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29316.09 MB 2025-02-14 19:49:05,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:49:05,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25371.65 MB 2025-02-14 19:49:05,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:49:05,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:49:05,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:49:05,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:05,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23954.22 MB 2025-02-14 19:49:05,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26196.08 MB 2025-02-14 19:49:05,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:49:05,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29316.09 MB 2025-02-14 19:49:05,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33562.82 MB 2025-02-14 19:49:05,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 19:49:05,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31740.36 MB 2025-02-14 19:49:05,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:49:05,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:49:05,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:49:05,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:05,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22064.69 MB 2025-02-14 19:49:05,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26196.08 MB 2025-02-14 19:49:05,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:49:05,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29316.09 MB 2025-02-14 19:49:05,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33562.82 MB 2025-02-14 19:49:05,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 19:49:05,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31740.36 MB 2025-02-14 19:49:05,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:49:05,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:49:05,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:49:05,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:05,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27729.62 MB 2025-02-14 19:49:05,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28496.62 MB 2025-02-14 19:49:05,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:49:05,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33562.82 MB 2025-02-14 19:49:05,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33978.06 MB 2025-02-14 19:49:05,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:49:05,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29204.41 MB 2025-02-14 19:49:05,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:49:05,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:49:05,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:49:05,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:05,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28909.51 MB 2025-02-14 19:49:05,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29139.75 MB 2025-02-14 19:49:05,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.24 MB 2025-02-14 19:49:05,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33978.06 MB 2025-02-14 19:49:05,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33978.06 MB 2025-02-14 19:49:05,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:49:05,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29322.85 MB 2025-02-14 19:49:05,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:49:05,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:49:05,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.49 seconds 2025-02-14 19:49:05,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:05,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16825.58 MB 2025-02-14 19:49:05,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29340.82 MB 2025-02-14 19:49:05,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12515.24 MB 2025-02-14 19:49:05,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53049.56 MB 2025-02-14 19:49:05,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33978.06 MB 2025-02-14 19:49:05,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19071.50 MB 2025-02-14 19:49:05,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29340.82 MB 2025-02-14 19:49:06,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:49:06,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:49:06,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:49:06,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:06,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29340.82 MB 2025-02-14 19:49:06,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21829.97 MB 2025-02-14 19:49:06,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7510.85 MB 2025-02-14 19:49:06,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33978.06 MB 2025-02-14 19:49:06,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33978.06 MB 2025-02-14 19:49:06,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:49:06,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31852.49 MB 2025-02-14 19:49:06,266 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:49:06,266 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:49:06,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:49:06,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:49:06,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:49:06,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:49:06,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21829.97 MB 2025-02-14 19:49:06,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30269.00 MB 2025-02-14 19:49:06,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:49:06,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33978.06 MB 2025-02-14 19:49:06,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42368.76 MB 2025-02-14 19:49:06,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:49:06,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30269.00 MB 2025-02-14 19:49:06,432 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:49:06,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:49:06,433 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:49:06,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:49:06,434 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:49:06,439 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:49:06,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:49:06,440 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:49:06,440 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:49:38,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:49:38,922 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:49:38,926 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:49:38,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:49:38,930 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2369, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:49:38,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:49:38,931 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2369, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:50:15,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:50:15,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:50:15,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.52 seconds 2025-02-14 19:50:15,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:15,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29476.28 MB 2025-02-14 19:50:15,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37860.69 MB 2025-02-14 19:50:15,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8384.41 MB 2025-02-14 19:50:15,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54953.77 MB 2025-02-14 19:50:15,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43255.86 MB 2025-02-14 19:50:15,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11697.91 MB 2025-02-14 19:50:15,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46874.88 MB 2025-02-14 19:50:15,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:50:15,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:50:15,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 19:50:15,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:15,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37860.69 MB 2025-02-14 19:50:15,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28093.54 MB 2025-02-14 19:50:15,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9767.15 MB 2025-02-14 19:50:15,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43255.86 MB 2025-02-14 19:50:15,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 68396.52 MB 2025-02-14 19:50:15,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25140.66 MB 2025-02-14 19:50:15,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59419.30 MB 2025-02-14 19:50:17,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:50:17,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:50:17,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 19:50:17,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:17,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28093.54 MB 2025-02-14 19:50:17,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28624.38 MB 2025-02-14 19:50:17,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:50:17,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68396.52 MB 2025-02-14 19:50:17,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32092.72 MB 2025-02-14 19:50:17,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36303.80 MB 2025-02-14 19:50:17,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32603.97 MB 2025-02-14 19:50:17,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:50:17,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:50:17,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:50:17,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:17,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28624.38 MB 2025-02-14 19:50:17,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30513.92 MB 2025-02-14 19:50:17,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:50:17,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 19:50:17,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33980.15 MB 2025-02-14 19:50:17,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:50:17,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31931.35 MB 2025-02-14 19:50:17,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:50:17,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:50:17,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:50:17,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:17,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30513.92 MB 2025-02-14 19:50:17,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32755.77 MB 2025-02-14 19:50:17,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:50:17,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33980.15 MB 2025-02-14 19:50:17,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40114.32 MB 2025-02-14 19:50:17,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:50:17,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38300.06 MB 2025-02-14 19:50:17,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:50:17,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:50:17,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:50:17,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:17,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28624.38 MB 2025-02-14 19:50:17,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32755.77 MB 2025-02-14 19:50:17,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:50:17,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32092.72 MB 2025-02-14 19:50:17,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40114.32 MB 2025-02-14 19:50:17,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 19:50:17,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38300.06 MB 2025-02-14 19:50:17,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:50:17,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:50:17,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:50:17,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:17,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34289.32 MB 2025-02-14 19:50:17,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35056.32 MB 2025-02-14 19:50:17,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:50:17,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40114.32 MB 2025-02-14 19:50:17,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40529.56 MB 2025-02-14 19:50:17,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:50:17,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35764.11 MB 2025-02-14 19:50:17,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:50:17,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:50:17,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:50:17,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:17,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35469.21 MB 2025-02-14 19:50:17,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35697.55 MB 2025-02-14 19:50:17,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 19:50:17,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40529.56 MB 2025-02-14 19:50:17,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40529.56 MB 2025-02-14 19:50:17,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:17,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35921.41 MB 2025-02-14 19:50:17,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:50:17,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:50:17,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.05 seconds 2025-02-14 19:50:17,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:17,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21222.49 MB 2025-02-14 19:50:17,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35897.94 MB 2025-02-14 19:50:17,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14675.45 MB 2025-02-14 19:50:17,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54953.77 MB 2025-02-14 19:50:17,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40529.56 MB 2025-02-14 19:50:17,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14424.21 MB 2025-02-14 19:50:17,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35921.41 MB 2025-02-14 19:50:18,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:50:18,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:50:18,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:50:18,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:18,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35897.94 MB 2025-02-14 19:50:18,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26216.31 MB 2025-02-14 19:50:18,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9681.62 MB 2025-02-14 19:50:18,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40529.56 MB 2025-02-14 19:50:18,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40529.56 MB 2025-02-14 19:50:18,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:18,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38401.10 MB 2025-02-14 19:50:18,268 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 19:50:18,268 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:50:18,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:50:18,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:50:18,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:50:18,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:18,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26216.31 MB 2025-02-14 19:50:18,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34626.12 MB 2025-02-14 19:50:18,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 19:50:18,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40529.56 MB 2025-02-14 19:50:18,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48890.90 MB 2025-02-14 19:50:18,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 19:50:18,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34626.12 MB 2025-02-14 19:50:18,430 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 19:50:18,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:18,431 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:50:18,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:18,432 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:50:18,437 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:50:18,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:18,438 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:50:18,438 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:50:30,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:30,101 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:50:30,105 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:50:30,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:30,109 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 733, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:50:30,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:30,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 733, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:50:41,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:50:41,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:50:41,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.50 seconds 2025-02-14 19:50:41,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:41,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18076.37 MB 2025-02-14 19:50:41,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20670.54 MB 2025-02-14 19:50:41,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2594.18 MB 2025-02-14 19:50:41,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61431.87 MB 2025-02-14 19:50:41,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22124.95 MB 2025-02-14 19:50:41,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39306.92 MB 2025-02-14 19:50:41,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29586.98 MB 2025-02-14 19:50:41,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:50:41,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:50:41,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 19:50:41,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:41,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20670.54 MB 2025-02-14 19:50:41,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19589.54 MB 2025-02-14 19:50:41,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1081.01 MB 2025-02-14 19:50:41,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22124.95 MB 2025-02-14 19:50:41,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35303.46 MB 2025-02-14 19:50:41,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13178.50 MB 2025-02-14 19:50:41,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29640.99 MB 2025-02-14 19:50:43,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:50:43,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:50:43,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 19:50:43,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:43,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19589.54 MB 2025-02-14 19:50:43,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20120.38 MB 2025-02-14 19:50:43,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:50:43,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35303.46 MB 2025-02-14 19:50:43,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22364.03 MB 2025-02-14 19:50:43,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12939.43 MB 2025-02-14 19:50:43,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24099.96 MB 2025-02-14 19:50:43,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:50:43,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:50:43,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:50:43,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:43,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20120.38 MB 2025-02-14 19:50:43,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22009.91 MB 2025-02-14 19:50:43,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:50:43,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22364.03 MB 2025-02-14 19:50:43,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25195.18 MB 2025-02-14 19:50:43,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 19:50:43,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23427.34 MB 2025-02-14 19:50:43,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:50:43,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:50:43,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:50:43,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:43,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22009.91 MB 2025-02-14 19:50:43,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24251.77 MB 2025-02-14 19:50:43,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:50:43,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25195.18 MB 2025-02-14 19:50:43,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31801.21 MB 2025-02-14 19:50:43,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 19:50:43,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29796.64 MB 2025-02-14 19:50:43,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:50:43,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:50:43,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 19:50:43,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:43,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20120.38 MB 2025-02-14 19:50:43,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24251.77 MB 2025-02-14 19:50:43,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:50:43,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22364.03 MB 2025-02-14 19:50:43,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31801.21 MB 2025-02-14 19:50:43,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 19:50:43,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29796.64 MB 2025-02-14 19:50:44,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:50:44,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:50:44,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:50:44,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:44,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25785.90 MB 2025-02-14 19:50:44,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26552.90 MB 2025-02-14 19:50:44,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:50:44,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31801.21 MB 2025-02-14 19:50:44,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32216.45 MB 2025-02-14 19:50:44,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:50:44,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27260.69 MB 2025-02-14 19:50:44,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:50:44,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:50:44,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:50:44,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:44,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26965.79 MB 2025-02-14 19:50:44,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27192.25 MB 2025-02-14 19:50:44,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.45 MB 2025-02-14 19:50:44,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32216.45 MB 2025-02-14 19:50:44,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32216.45 MB 2025-02-14 19:50:44,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:44,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27378.84 MB 2025-02-14 19:50:44,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:50:44,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:50:44,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.92 seconds 2025-02-14 19:50:44,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:44,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15522.54 MB 2025-02-14 19:50:44,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27393.02 MB 2025-02-14 19:50:44,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11870.49 MB 2025-02-14 19:50:44,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61431.87 MB 2025-02-14 19:50:44,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32216.45 MB 2025-02-14 19:50:44,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29215.42 MB 2025-02-14 19:50:44,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27393.02 MB 2025-02-14 19:50:44,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:50:44,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:50:44,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:50:44,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:44,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27393.02 MB 2025-02-14 19:50:44,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20522.95 MB 2025-02-14 19:50:44,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6870.08 MB 2025-02-14 19:50:44,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32216.45 MB 2025-02-14 19:50:44,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32216.45 MB 2025-02-14 19:50:44,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:44,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29901.00 MB 2025-02-14 19:50:44,324 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 19:50:44,325 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:50:44,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:50:44,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:50:44,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:50:44,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:44,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20522.95 MB 2025-02-14 19:50:44,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28949.45 MB 2025-02-14 19:50:44,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 19:50:44,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32216.45 MB 2025-02-14 19:50:44,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42689.63 MB 2025-02-14 19:50:44,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 19:50:44,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28949.45 MB 2025-02-14 19:50:44,489 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 19:50:44,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:44,490 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:50:44,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:44,491 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:50:44,496 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:50:44,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:44,497 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:50:44,497 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:50:51,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:51,213 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:50:51,218 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:50:51,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:51,222 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 224, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:50:51,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:51,223 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 224, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:50:54,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:50:54,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:50:54,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.51 seconds 2025-02-14 19:50:54,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:54,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14529.57 MB 2025-02-14 19:50:54,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15322.30 MB 2025-02-14 19:50:54,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 792.72 MB 2025-02-14 19:50:54,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55255.76 MB 2025-02-14 19:50:54,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 19:50:54,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31977.37 MB 2025-02-14 19:50:54,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24227.44 MB 2025-02-14 19:50:54,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:50:54,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:50:54,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:50:54,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:54,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15322.30 MB 2025-02-14 19:50:54,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15376.29 MB 2025-02-14 19:50:54,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 53.99 MB 2025-02-14 19:50:54,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 19:50:54,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 19:50:54,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:54,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17865.14 MB 2025-02-14 19:50:55,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:50:55,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:50:55,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 19:50:55,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:55,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15376.29 MB 2025-02-14 19:50:55,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15611.19 MB 2025-02-14 19:50:55,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 19:50:55,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 19:50:55,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 19:50:55,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 19:50:55,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19545.94 MB 2025-02-14 19:50:55,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:50:55,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:50:55,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:50:55,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:55,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15611.12 MB 2025-02-14 19:50:55,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16447.04 MB 2025-02-14 19:50:55,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 19:50:55,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 19:50:55,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 19:50:55,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:55,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17074.25 MB 2025-02-14 19:50:55,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:50:55,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:50:55,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 19:50:55,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:55,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16447.04 MB 2025-02-14 19:50:55,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17439.09 MB 2025-02-14 19:50:55,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 19:50:55,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 19:50:55,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 19:50:55,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:55,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19892.40 MB 2025-02-14 19:50:55,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:50:55,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:50:55,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 19:50:55,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:55,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15611.12 MB 2025-02-14 19:50:55,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17439.09 MB 2025-02-14 19:50:55,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 19:50:55,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 19:50:55,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 19:50:55,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:55,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19892.40 MB 2025-02-14 19:50:55,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:50:55,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:50:55,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 19:50:55,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:55,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18117.69 MB 2025-02-14 19:50:55,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18457.08 MB 2025-02-14 19:50:55,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 19:50:55,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 19:50:55,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 19:50:55,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 19:50:55,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18776.91 MB 2025-02-14 19:50:55,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:50:55,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:50:55,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:50:55,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:55,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18639.79 MB 2025-02-14 19:50:55,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18869.05 MB 2025-02-14 19:50:55,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.25 MB 2025-02-14 19:50:55,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-14 19:50:55,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 19:50:55,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:55,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18892.04 MB 2025-02-14 19:50:55,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:50:55,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:50:55,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.57 seconds 2025-02-14 19:50:55,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:55,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13749.14 MB 2025-02-14 19:50:55,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19070.12 MB 2025-02-14 19:50:55,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5320.98 MB 2025-02-14 19:50:55,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55255.76 MB 2025-02-14 19:50:55,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 19:50:55,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32268.88 MB 2025-02-14 19:50:55,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19070.12 MB 2025-02-14 19:50:56,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:50:56,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:50:56,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:50:56,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:56,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19070.12 MB 2025-02-14 19:50:56,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17701.13 MB 2025-02-14 19:50:56,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1368.99 MB 2025-02-14 19:50:56,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-14 19:50:56,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22986.88 MB 2025-02-14 19:50:56,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:50:56,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19304.54 MB 2025-02-14 19:50:56,087 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 19:50:56,087 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:50:56,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:50:56,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:50:56,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:50:56,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:50:56,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17701.13 MB 2025-02-14 19:50:56,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26140.15 MB 2025-02-14 19:50:56,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 19:50:56,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22986.88 MB 2025-02-14 19:50:56,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31377.59 MB 2025-02-14 19:50:56,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 19:50:56,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26140.15 MB 2025-02-14 19:50:56,253 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 19:50:56,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:56,254 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:50:56,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:56,255 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:50:56,260 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:50:56,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:50:56,261 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:50:56,261 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:51:07,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:07,054 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:51:07,059 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:51:07,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:07,062 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 115, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:51:07,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:07,063 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 115, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:51:08,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:51:08,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:51:08,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.80 seconds 2025-02-14 19:51:08,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:08,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13770.05 MB 2025-02-14 19:51:08,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14177.02 MB 2025-02-14 19:51:08,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.98 MB 2025-02-14 19:51:08,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43962.60 MB 2025-02-14 19:51:08,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 19:51:08,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25375.54 MB 2025-02-14 19:51:08,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23014.92 MB 2025-02-14 19:51:08,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:51:08,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:51:08,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:51:08,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:08,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14177.02 MB 2025-02-14 19:51:08,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14374.20 MB 2025-02-14 19:51:08,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.18 MB 2025-02-14 19:51:08,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 19:51:08,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 19:51:08,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:08,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14984.74 MB 2025-02-14 19:51:09,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:51:09,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:51:09,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.56 seconds 2025-02-14 19:51:09,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14374.20 MB 2025-02-14 19:51:09,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14526.82 MB 2025-02-14 19:51:09,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 152.62 MB 2025-02-14 19:51:09,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 19:51:09,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 19:51:09,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:09,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18459.96 MB 2025-02-14 19:51:09,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:51:09,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:51:09,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:51:09,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14526.76 MB 2025-02-14 19:51:09,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15069.86 MB 2025-02-14 19:51:09,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 543.11 MB 2025-02-14 19:51:09,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 19:51:09,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 19:51:09,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:09,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15477.38 MB 2025-02-14 19:51:09,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:51:09,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:51:09,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:51:09,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15069.86 MB 2025-02-14 19:51:09,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15729.52 MB 2025-02-14 19:51:09,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 659.65 MB 2025-02-14 19:51:09,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 19:51:09,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 19:51:09,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:09,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17308.38 MB 2025-02-14 19:51:09,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:51:09,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:51:09,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:51:09,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14526.76 MB 2025-02-14 19:51:09,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15729.52 MB 2025-02-14 19:51:09,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1202.76 MB 2025-02-14 19:51:09,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 19:51:09,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 19:51:09,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:09,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17308.38 MB 2025-02-14 19:51:09,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:51:09,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:51:09,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:51:09,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16366.36 MB 2025-02-14 19:51:09,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16643.40 MB 2025-02-14 19:51:09,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.04 MB 2025-02-14 19:51:09,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 19:51:09,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18763.22 MB 2025-02-14 19:51:09,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 176.16 MB 2025-02-14 19:51:09,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16846.89 MB 2025-02-14 19:51:09,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:51:09,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:51:09,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:51:09,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16818.64 MB 2025-02-14 19:51:09,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17048.25 MB 2025-02-14 19:51:09,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.61 MB 2025-02-14 19:51:09,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18763.22 MB 2025-02-14 19:51:09,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18763.22 MB 2025-02-14 19:51:09,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:09,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17048.25 MB 2025-02-14 19:51:09,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:51:09,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:51:09,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.56 seconds 2025-02-14 19:51:09,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13369.38 MB 2025-02-14 19:51:09,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17248.36 MB 2025-02-14 19:51:09,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3878.99 MB 2025-02-14 19:51:09,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43962.60 MB 2025-02-14 19:51:09,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18763.22 MB 2025-02-14 19:51:09,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25199.38 MB 2025-02-14 19:51:09,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17248.36 MB 2025-02-14 19:51:09,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:51:09,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:51:09,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:51:09,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17248.36 MB 2025-02-14 19:51:09,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20248.02 MB 2025-02-14 19:51:09,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2999.66 MB 2025-02-14 19:51:09,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18763.22 MB 2025-02-14 19:51:09,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21716.01 MB 2025-02-14 19:51:09,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2952.79 MB 2025-02-14 19:51:09,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20548.85 MB 2025-02-14 19:51:09,910 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 19:51:09,910 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 19:51:09,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:51:09,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:51:09,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:51:09,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:09,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20248.02 MB 2025-02-14 19:51:09,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28647.41 MB 2025-02-14 19:51:09,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-14 19:51:09,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21716.01 MB 2025-02-14 19:51:09,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32155.63 MB 2025-02-14 19:51:09,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10439.62 MB 2025-02-14 19:51:09,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28647.41 MB 2025-02-14 19:51:10,079 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 19:51:10,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:10,081 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:51:10,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:10,082 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:51:10,086 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:51:10,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:10,088 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:51:10,088 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 19:51:25,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:25,131 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:51:25,136 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:51:25,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:25,140 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 157, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:51:25,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:25,141 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 157, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:51:27,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:51:27,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:51:27,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.44 seconds 2025-02-14 19:51:27,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:27,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22398.84 MB 2025-02-14 19:51:27,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22954.46 MB 2025-02-14 19:51:27,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 555.61 MB 2025-02-14 19:51:27,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40508.59 MB 2025-02-14 19:51:27,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25769.80 MB 2025-02-14 19:51:27,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14738.78 MB 2025-02-14 19:51:27,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31871.02 MB 2025-02-14 19:51:27,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:51:27,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:51:27,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:51:27,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:27,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22954.46 MB 2025-02-14 19:51:27,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23223.78 MB 2025-02-14 19:51:27,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 269.32 MB 2025-02-14 19:51:27,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25769.80 MB 2025-02-14 19:51:27,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26325.55 MB 2025-02-14 19:51:27,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 555.75 MB 2025-02-14 19:51:27,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25205.89 MB 2025-02-14 19:51:28,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:51:28,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:51:28,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.76 seconds 2025-02-14 19:51:28,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23223.78 MB 2025-02-14 19:51:28,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23432.14 MB 2025-02-14 19:51:28,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.36 MB 2025-02-14 19:51:28,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26325.55 MB 2025-02-14 19:51:28,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25583.16 MB 2025-02-14 19:51:28,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -742.39 MB 2025-02-14 19:51:28,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27394.47 MB 2025-02-14 19:51:28,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:51:28,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:51:28,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:51:28,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23432.07 MB 2025-02-14 19:51:28,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24173.53 MB 2025-02-14 19:51:28,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 741.46 MB 2025-02-14 19:51:28,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25583.16 MB 2025-02-14 19:51:28,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25585.25 MB 2025-02-14 19:51:28,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 19:51:28,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24729.88 MB 2025-02-14 19:51:28,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:51:28,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:51:28,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:51:28,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24173.53 MB 2025-02-14 19:51:28,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25054.64 MB 2025-02-14 19:51:28,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 881.11 MB 2025-02-14 19:51:28,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25585.25 MB 2025-02-14 19:51:28,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 19:51:28,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2600.47 MB 2025-02-14 19:51:28,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27233.88 MB 2025-02-14 19:51:28,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:51:28,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:51:28,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:51:28,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23432.07 MB 2025-02-14 19:51:28,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25054.64 MB 2025-02-14 19:51:28,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1622.57 MB 2025-02-14 19:51:28,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25583.16 MB 2025-02-14 19:51:28,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 19:51:28,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2602.57 MB 2025-02-14 19:51:28,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27233.88 MB 2025-02-14 19:51:28,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:51:28,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:51:28,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 19:51:28,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25656.56 MB 2025-02-14 19:51:28,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17621.47 MB 2025-02-14 19:51:28,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8035.09 MB 2025-02-14 19:51:28,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28185.72 MB 2025-02-14 19:51:28,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 19:51:28,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:28,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25858.54 MB 2025-02-14 19:51:28,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:51:28,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:51:28,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:51:28,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17783.54 MB 2025-02-14 19:51:28,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17998.94 MB 2025-02-14 19:51:28,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 215.40 MB 2025-02-14 19:51:28,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28185.72 MB 2025-02-14 19:51:28,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 19:51:28,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:28,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18019.57 MB 2025-02-14 19:51:28,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:51:28,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:51:28,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-14 19:51:28,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21851.84 MB 2025-02-14 19:51:28,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18199.72 MB 2025-02-14 19:51:28,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3652.12 MB 2025-02-14 19:51:28,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40508.59 MB 2025-02-14 19:51:28,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 19:51:28,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12322.87 MB 2025-02-14 19:51:28,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18199.72 MB 2025-02-14 19:51:28,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:51:28,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:51:28,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:51:28,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18199.72 MB 2025-02-14 19:51:28,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17368.74 MB 2025-02-14 19:51:28,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -830.99 MB 2025-02-14 19:51:28,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28185.72 MB 2025-02-14 19:51:28,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 19:51:28,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:51:28,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19002.28 MB 2025-02-14 19:51:28,815 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 19:51:28,815 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 19:51:28,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:51:28,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:51:28,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:51:28,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:51:28,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17368.74 MB 2025-02-14 19:51:28,821 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25795.24 MB 2025-02-14 19:51:28,821 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 19:51:28,821 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28185.72 MB 2025-02-14 19:51:28,821 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36563.85 MB 2025-02-14 19:51:28,821 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 19:51:28,821 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25795.24 MB 2025-02-14 19:51:28,979 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 19:51:28,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:28,980 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:51:28,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:28,981 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:51:28,986 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:51:28,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:51:28,987 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:51:28,987 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 19:53:14,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:53:14,959 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:53:14,966 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:53:14,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:53:14,972 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 321, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:53:14,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:53:14,974 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 321, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:53:19,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:53:19,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:53:19,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.96 seconds 2025-02-14 19:53:19,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:19,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15205.49 MB 2025-02-14 19:53:19,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16341.49 MB 2025-02-14 19:53:19,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1136.00 MB 2025-02-14 19:53:19,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49129.98 MB 2025-02-14 19:53:19,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24033.36 MB 2025-02-14 19:53:19,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25096.62 MB 2025-02-14 19:53:19,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25356.33 MB 2025-02-14 19:53:19,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:53:19,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:53:19,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:53:19,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:19,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16341.49 MB 2025-02-14 19:53:19,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16660.05 MB 2025-02-14 19:53:19,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.56 MB 2025-02-14 19:53:19,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24033.36 MB 2025-02-14 19:53:19,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24033.36 MB 2025-02-14 19:53:19,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:53:19,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20390.29 MB 2025-02-14 19:53:21,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:53:21,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:53:21,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.36 seconds 2025-02-14 19:53:21,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16660.05 MB 2025-02-14 19:53:21,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17042.26 MB 2025-02-14 19:53:21,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 382.21 MB 2025-02-14 19:53:21,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24033.36 MB 2025-02-14 19:53:21,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24033.36 MB 2025-02-14 19:53:21,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:53:21,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20999.57 MB 2025-02-14 19:53:21,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:53:21,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:53:21,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:53:21,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17042.26 MB 2025-02-14 19:53:21,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18403.31 MB 2025-02-14 19:53:21,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1361.05 MB 2025-02-14 19:53:21,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24033.36 MB 2025-02-14 19:53:21,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24033.36 MB 2025-02-14 19:53:21,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:53:21,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19423.86 MB 2025-02-14 19:53:21,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:53:21,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:53:21,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 19:53:21,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18403.31 MB 2025-02-14 19:53:21,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20017.46 MB 2025-02-14 19:53:21,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1614.15 MB 2025-02-14 19:53:21,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24033.36 MB 2025-02-14 19:53:21,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25392.32 MB 2025-02-14 19:53:21,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1358.95 MB 2025-02-14 19:53:21,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24009.33 MB 2025-02-14 19:53:21,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:53:21,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:53:21,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:53:21,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17042.26 MB 2025-02-14 19:53:21,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20017.46 MB 2025-02-14 19:53:21,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2975.21 MB 2025-02-14 19:53:21,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24033.36 MB 2025-02-14 19:53:21,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25392.32 MB 2025-02-14 19:53:21,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1358.95 MB 2025-02-14 19:53:21,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24009.33 MB 2025-02-14 19:53:21,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:53:21,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:53:21,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 19:53:21,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21121.61 MB 2025-02-14 19:53:21,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21673.85 MB 2025-02-14 19:53:21,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.24 MB 2025-02-14 19:53:21,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25392.32 MB 2025-02-14 19:53:21,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25692.21 MB 2025-02-14 19:53:21,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 299.89 MB 2025-02-14 19:53:21,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22183.46 MB 2025-02-14 19:53:21,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:53:21,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:53:21,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:53:21,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21971.14 MB 2025-02-14 19:53:21,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22199.21 MB 2025-02-14 19:53:21,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.07 MB 2025-02-14 19:53:21,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25692.21 MB 2025-02-14 19:53:21,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25692.21 MB 2025-02-14 19:53:21,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:53:21,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22285.01 MB 2025-02-14 19:53:21,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:53:21,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:53:21,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.65 seconds 2025-02-14 19:53:21,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14087.10 MB 2025-02-14 19:53:21,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22399.89 MB 2025-02-14 19:53:21,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8312.80 MB 2025-02-14 19:53:21,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49129.98 MB 2025-02-14 19:53:21,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25692.21 MB 2025-02-14 19:53:21,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23437.77 MB 2025-02-14 19:53:21,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22399.89 MB 2025-02-14 19:53:21,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:53:21,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:53:21,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 19:53:21,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22399.89 MB 2025-02-14 19:53:21,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25408.03 MB 2025-02-14 19:53:21,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.14 MB 2025-02-14 19:53:21,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25692.21 MB 2025-02-14 19:53:21,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26765.95 MB 2025-02-14 19:53:21,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1073.74 MB 2025-02-14 19:53:21,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25709.33 MB 2025-02-14 19:53:21,922 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 19:53:21,923 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:53:21,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:53:21,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:53:21,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:53:21,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:53:21,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18556.83 MB 2025-02-14 19:53:21,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26979.16 MB 2025-02-14 19:53:21,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 19:53:21,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26765.95 MB 2025-02-14 19:53:21,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35139.88 MB 2025-02-14 19:53:21,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 19:53:21,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26979.16 MB 2025-02-14 19:53:22,090 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 19:53:22,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:53:22,091 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:53:22,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:53:22,092 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:53:22,097 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:53:22,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:53:22,098 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:53:22,098 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:54:57,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:54:57,764 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:54:57,772 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:54:57,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:54:57,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2489, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:54:57,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:54:57,781 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2489, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:55:35,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:55:35,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:55:35,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.13 seconds 2025-02-14 19:55:35,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:35,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30312.55 MB 2025-02-14 19:55:35,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39120.98 MB 2025-02-14 19:55:35,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8808.43 MB 2025-02-14 19:55:35,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65047.36 MB 2025-02-14 19:55:35,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44272.98 MB 2025-02-14 19:55:35,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20774.39 MB 2025-02-14 19:55:35,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47937.64 MB 2025-02-14 19:55:36,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:55:36,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:55:36,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:55:36,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:36,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39120.98 MB 2025-02-14 19:55:36,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28717.47 MB 2025-02-14 19:55:36,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10403.50 MB 2025-02-14 19:55:36,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44272.98 MB 2025-02-14 19:55:36,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 74637.64 MB 2025-02-14 19:55:36,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30364.66 MB 2025-02-14 19:55:36,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64439.70 MB 2025-02-14 19:55:38,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:55:38,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:55:38,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 19:55:38,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28717.47 MB 2025-02-14 19:55:38,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29248.32 MB 2025-02-14 19:55:38,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:55:38,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74637.64 MB 2025-02-14 19:55:38,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32472.30 MB 2025-02-14 19:55:38,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42165.34 MB 2025-02-14 19:55:38,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33227.90 MB 2025-02-14 19:55:38,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:55:38,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:55:38,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:55:38,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29248.32 MB 2025-02-14 19:55:38,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31137.65 MB 2025-02-14 19:55:38,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.34 MB 2025-02-14 19:55:38,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32472.30 MB 2025-02-14 19:55:38,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34359.74 MB 2025-02-14 19:55:38,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:55:38,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.08 MB 2025-02-14 19:55:38,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:55:38,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:55:38,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:55:38,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31137.65 MB 2025-02-14 19:55:38,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33379.51 MB 2025-02-14 19:55:38,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:55:38,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34359.74 MB 2025-02-14 19:55:38,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40493.91 MB 2025-02-14 19:55:38,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 19:55:38,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38923.79 MB 2025-02-14 19:55:38,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:55:38,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:55:38,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:55:38,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29248.32 MB 2025-02-14 19:55:38,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33379.51 MB 2025-02-14 19:55:38,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.19 MB 2025-02-14 19:55:38,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32472.30 MB 2025-02-14 19:55:38,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40493.91 MB 2025-02-14 19:55:38,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 19:55:38,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38923.79 MB 2025-02-14 19:55:38,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:55:38,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:55:38,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:55:38,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34913.05 MB 2025-02-14 19:55:38,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35680.05 MB 2025-02-14 19:55:38,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:55:38,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40493.91 MB 2025-02-14 19:55:38,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40909.14 MB 2025-02-14 19:55:38,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 19:55:38,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36387.84 MB 2025-02-14 19:55:38,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:55:38,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:55:38,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:55:38,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36092.94 MB 2025-02-14 19:55:38,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36322.08 MB 2025-02-14 19:55:38,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-14 19:55:38,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40909.14 MB 2025-02-14 19:55:38,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40909.14 MB 2025-02-14 19:55:38,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:55:38,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36551.54 MB 2025-02-14 19:55:38,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:55:38,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:55:38,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.68 seconds 2025-02-14 19:55:38,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21640.63 MB 2025-02-14 19:55:38,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36523.12 MB 2025-02-14 19:55:38,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14882.50 MB 2025-02-14 19:55:38,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56373.54 MB 2025-02-14 19:55:38,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40909.14 MB 2025-02-14 19:55:38,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15464.40 MB 2025-02-14 19:55:38,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36551.54 MB 2025-02-14 19:55:38,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:55:38,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:55:38,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:55:38,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36523.12 MB 2025-02-14 19:55:38,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26644.63 MB 2025-02-14 19:55:38,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9878.49 MB 2025-02-14 19:55:38,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40909.14 MB 2025-02-14 19:55:38,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40909.14 MB 2025-02-14 19:55:38,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:55:38,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39034.48 MB 2025-02-14 19:55:38,755 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 19:55:38,756 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:55:38,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:55:38,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:55:38,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:55:38,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:55:38,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26644.63 MB 2025-02-14 19:55:38,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35083.14 MB 2025-02-14 19:55:38,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.50 MB 2025-02-14 19:55:38,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40909.14 MB 2025-02-14 19:55:38,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45103.45 MB 2025-02-14 19:55:38,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 19:55:38,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35083.14 MB 2025-02-14 19:55:38,928 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 19:55:38,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:55:38,930 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:55:38,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:55:38,931 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:55:38,936 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:55:38,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:55:38,937 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:55:38,937 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:56:32,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:56:32,348 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:56:32,353 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:56:32,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:56:32,357 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1801, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:56:32,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:56:32,358 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1801, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:57:00,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:57:00,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:57:00,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.84 seconds 2025-02-14 19:57:00,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:00,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25518.36 MB 2025-02-14 19:57:00,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31892.00 MB 2025-02-14 19:57:00,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6373.64 MB 2025-02-14 19:57:00,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53492.06 MB 2025-02-14 19:57:00,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37075.55 MB 2025-02-14 19:57:00,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16416.51 MB 2025-02-14 19:57:00,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40879.34 MB 2025-02-14 19:57:00,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:57:00,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:57:00,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 19:57:00,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:00,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31892.00 MB 2025-02-14 19:57:00,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25140.69 MB 2025-02-14 19:57:00,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6751.31 MB 2025-02-14 19:57:00,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37075.55 MB 2025-02-14 19:57:00,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59894.66 MB 2025-02-14 19:57:00,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22819.11 MB 2025-02-14 19:57:00,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50566.09 MB 2025-02-14 19:57:02,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:57:02,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:57:02,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 19:57:02,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25140.69 MB 2025-02-14 19:57:02,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.53 MB 2025-02-14 19:57:02,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:57:02,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59894.66 MB 2025-02-14 19:57:02,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32115.79 MB 2025-02-14 19:57:02,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27778.88 MB 2025-02-14 19:57:02,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29650.08 MB 2025-02-14 19:57:02,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:57:02,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:57:02,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:57:02,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-14 19:57:02,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27561.06 MB 2025-02-14 19:57:02,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:57:02,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 19:57:02,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32115.79 MB 2025-02-14 19:57:02,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:57:02,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28978.49 MB 2025-02-14 19:57:02,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:57:02,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:57:02,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:57:02,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27561.06 MB 2025-02-14 19:57:02,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-14 19:57:02,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:57:02,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 19:57:02,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37306.24 MB 2025-02-14 19:57:02,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 19:57:02,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-14 19:57:02,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:57:02,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:57:02,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:57:02,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-14 19:57:02,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-14 19:57:02,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:57:02,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32115.79 MB 2025-02-14 19:57:02,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37306.24 MB 2025-02-14 19:57:02,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 19:57:02,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-14 19:57:02,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:57:02,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:57:02,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:57:02,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31336.46 MB 2025-02-14 19:57:02,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32103.46 MB 2025-02-14 19:57:02,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:57:02,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37306.24 MB 2025-02-14 19:57:02,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37719.38 MB 2025-02-14 19:57:02,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 19:57:02,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32811.25 MB 2025-02-14 19:57:02,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:57:02,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:57:02,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:57:02,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32516.35 MB 2025-02-14 19:57:02,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32744.22 MB 2025-02-14 19:57:02,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.87 MB 2025-02-14 19:57:02,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37719.38 MB 2025-02-14 19:57:02,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37719.38 MB 2025-02-14 19:57:02,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:57:02,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.88 MB 2025-02-14 19:57:02,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:57:02,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:57:02,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.32 seconds 2025-02-14 19:57:02,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19243.53 MB 2025-02-14 19:57:02,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32944.34 MB 2025-02-14 19:57:02,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13700.80 MB 2025-02-14 19:57:02,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53492.06 MB 2025-02-14 19:57:02,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37719.38 MB 2025-02-14 19:57:02,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15772.68 MB 2025-02-14 19:57:02,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.88 MB 2025-02-14 19:57:02,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:57:02,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:57:02,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:57:02,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32944.34 MB 2025-02-14 19:57:02,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24233.11 MB 2025-02-14 19:57:02,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8711.23 MB 2025-02-14 19:57:02,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37719.38 MB 2025-02-14 19:57:02,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37719.38 MB 2025-02-14 19:57:02,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:57:02,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35444.39 MB 2025-02-14 19:57:02,962 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 19:57:02,962 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:57:02,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:57:02,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:57:02,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:57:02,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:02,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24233.11 MB 2025-02-14 19:57:02,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32631.54 MB 2025-02-14 19:57:02,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8398.43 MB 2025-02-14 19:57:02,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37719.38 MB 2025-02-14 19:57:02,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41894.81 MB 2025-02-14 19:57:02,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 19:57:02,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32631.54 MB 2025-02-14 19:57:03,129 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 19:57:03,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:03,130 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:57:03,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:03,131 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:57:03,136 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:57:03,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:03,137 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:57:03,137 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:57:36,496 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:36,496 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:57:36,501 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:57:36,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:36,506 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1255, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:57:36,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:36,507 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1255, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:57:55,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:57:55,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:57:55,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.43 seconds 2025-02-14 19:57:55,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:55,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21713.75 MB 2025-02-14 19:57:55,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26155.51 MB 2025-02-14 19:57:55,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4441.77 MB 2025-02-14 19:57:55,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50245.66 MB 2025-02-14 19:57:55,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35081.16 MB 2025-02-14 19:57:55,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15164.51 MB 2025-02-14 19:57:55,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35035.49 MB 2025-02-14 19:57:56,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:57:56,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:57:56,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 19:57:56,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:56,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26155.51 MB 2025-02-14 19:57:56,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22302.20 MB 2025-02-14 19:57:56,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3853.31 MB 2025-02-14 19:57:56,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35081.16 MB 2025-02-14 19:57:56,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43771.76 MB 2025-02-14 19:57:56,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8690.60 MB 2025-02-14 19:57:56,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39249.25 MB 2025-02-14 19:57:57,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:57:57,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:57:57,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 19:57:57,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:57,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22302.20 MB 2025-02-14 19:57:57,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22833.05 MB 2025-02-14 19:57:57,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:57:57,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43771.76 MB 2025-02-14 19:57:57,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26463.96 MB 2025-02-14 19:57:57,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17307.80 MB 2025-02-14 19:57:57,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26812.63 MB 2025-02-14 19:57:57,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:57:57,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:57:57,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:57:57,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:57,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22833.05 MB 2025-02-14 19:57:57,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24722.58 MB 2025-02-14 19:57:57,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:57:57,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26463.96 MB 2025-02-14 19:57:57,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27407.68 MB 2025-02-14 19:57:57,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 19:57:57,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26140.01 MB 2025-02-14 19:57:58,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:57:58,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:57:58,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:57:58,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:58,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24722.58 MB 2025-02-14 19:57:58,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26964.44 MB 2025-02-14 19:57:58,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:57:58,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27407.68 MB 2025-02-14 19:57:58,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-14 19:57:58,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 19:57:58,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32508.72 MB 2025-02-14 19:57:58,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:57:58,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:57:58,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:57:58,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:58,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22833.05 MB 2025-02-14 19:57:58,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26964.44 MB 2025-02-14 19:57:58,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:57:58,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26463.96 MB 2025-02-14 19:57:58,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-14 19:57:58,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 19:57:58,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32508.72 MB 2025-02-14 19:57:58,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:57:58,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:57:58,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 19:57:58,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:58,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28497.98 MB 2025-02-14 19:57:58,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29264.98 MB 2025-02-14 19:57:58,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:57:58,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-14 19:57:58,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34431.04 MB 2025-02-14 19:57:58,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:57:58,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29972.77 MB 2025-02-14 19:57:58,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:57:58,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:57:58,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:57:58,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:58,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29677.87 MB 2025-02-14 19:57:58,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29907.12 MB 2025-02-14 19:57:58,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.26 MB 2025-02-14 19:57:58,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34431.04 MB 2025-02-14 19:57:58,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34431.04 MB 2025-02-14 19:57:58,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:57:58,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30144.05 MB 2025-02-14 19:57:58,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:57:58,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:57:58,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.86 seconds 2025-02-14 19:57:58,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:58,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17341.23 MB 2025-02-14 19:57:58,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30108.12 MB 2025-02-14 19:57:58,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12766.90 MB 2025-02-14 19:57:58,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50245.66 MB 2025-02-14 19:57:58,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34431.04 MB 2025-02-14 19:57:58,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15814.62 MB 2025-02-14 19:57:58,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30144.05 MB 2025-02-14 19:57:58,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:57:58,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:57:58,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:57:58,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:58,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30108.12 MB 2025-02-14 19:57:58,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22344.47 MB 2025-02-14 19:57:58,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7763.65 MB 2025-02-14 19:57:58,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34431.04 MB 2025-02-14 19:57:58,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34431.04 MB 2025-02-14 19:57:58,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:57:58,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32618.87 MB 2025-02-14 19:57:58,662 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 19:57:58,662 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:57:58,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:57:58,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:57:58,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:57:58,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:57:58,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.47 MB 2025-02-14 19:57:58,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30780.07 MB 2025-02-14 19:57:58,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 19:57:58,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34431.04 MB 2025-02-14 19:57:58,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42819.65 MB 2025-02-14 19:57:58,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 19:57:58,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30780.07 MB 2025-02-14 19:57:58,827 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 19:57:58,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:58,828 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:57:58,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:58,829 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:57:58,834 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:57:58,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:57:58,835 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:57:58,835 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:58:08,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:58:08,310 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:58:08,315 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:58:08,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:58:08,319 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 674, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:58:08,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:58:08,320 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 674, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:58:18,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:58:18,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:58:18,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.51 seconds 2025-02-14 19:58:18,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:18,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17665.25 MB 2025-02-14 19:58:18,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20050.49 MB 2025-02-14 19:58:18,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2385.25 MB 2025-02-14 19:58:18,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51208.26 MB 2025-02-14 19:58:18,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24675.09 MB 2025-02-14 19:58:18,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26533.17 MB 2025-02-14 19:58:18,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28949.36 MB 2025-02-14 19:58:18,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:58:18,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:58:18,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 19:58:18,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:18,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20050.49 MB 2025-02-14 19:58:18,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19281.77 MB 2025-02-14 19:58:18,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -768.73 MB 2025-02-14 19:58:18,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24675.09 MB 2025-02-14 19:58:18,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31868.32 MB 2025-02-14 19:58:18,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7193.23 MB 2025-02-14 19:58:18,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28644.43 MB 2025-02-14 19:58:20,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:58:20,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:58:20,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 19:58:20,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:20,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19281.77 MB 2025-02-14 19:58:20,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19812.61 MB 2025-02-14 19:58:20,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:58:20,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31868.32 MB 2025-02-14 19:58:20,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23704.11 MB 2025-02-14 19:58:20,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8164.21 MB 2025-02-14 19:58:20,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23792.19 MB 2025-02-14 19:58:20,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:58:20,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:58:20,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:58:20,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:20,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19812.61 MB 2025-02-14 19:58:20,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21702.14 MB 2025-02-14 19:58:20,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:58:20,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23704.11 MB 2025-02-14 19:58:20,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25591.55 MB 2025-02-14 19:58:20,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 19:58:20,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23119.57 MB 2025-02-14 19:58:21,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:58:21,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:58:21,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:58:21,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:21,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21702.14 MB 2025-02-14 19:58:21,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23944.00 MB 2025-02-14 19:58:21,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:58:21,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25591.55 MB 2025-02-14 19:58:21,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31253.86 MB 2025-02-14 19:58:21,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 19:58:21,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29488.28 MB 2025-02-14 19:58:21,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:58:21,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:58:21,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:58:21,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:21,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19812.61 MB 2025-02-14 19:58:21,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23944.00 MB 2025-02-14 19:58:21,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:58:21,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23704.11 MB 2025-02-14 19:58:21,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31253.86 MB 2025-02-14 19:58:21,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 19:58:21,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29488.28 MB 2025-02-14 19:58:21,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:58:21,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:58:21,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 19:58:21,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:21,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25477.54 MB 2025-02-14 19:58:21,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26244.54 MB 2025-02-14 19:58:21,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:58:21,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31253.86 MB 2025-02-14 19:58:21,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31671.19 MB 2025-02-14 19:58:21,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:58:21,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26952.33 MB 2025-02-14 19:58:21,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:58:21,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:58:21,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:58:21,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:21,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26657.43 MB 2025-02-14 19:58:21,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26886.07 MB 2025-02-14 19:58:21,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-14 19:58:21,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31671.19 MB 2025-02-14 19:58:21,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31671.19 MB 2025-02-14 19:58:21,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:58:21,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27094.88 MB 2025-02-14 19:58:21,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:58:21,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:58:21,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.89 seconds 2025-02-14 19:58:21,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:21,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15316.98 MB 2025-02-14 19:58:21,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27086.63 MB 2025-02-14 19:58:21,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11769.65 MB 2025-02-14 19:58:21,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51208.26 MB 2025-02-14 19:58:21,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31671.19 MB 2025-02-14 19:58:21,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19537.07 MB 2025-02-14 19:58:21,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27094.88 MB 2025-02-14 19:58:21,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:58:21,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:58:21,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:58:21,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:21,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27086.63 MB 2025-02-14 19:58:21,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20313.37 MB 2025-02-14 19:58:21,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6773.26 MB 2025-02-14 19:58:21,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31671.19 MB 2025-02-14 19:58:21,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31671.19 MB 2025-02-14 19:58:21,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:58:21,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29591.84 MB 2025-02-14 19:58:21,502 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 19:58:21,503 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:58:21,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:58:21,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:58:21,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:58:21,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:58:21,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20313.37 MB 2025-02-14 19:58:21,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28731.11 MB 2025-02-14 19:58:21,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-14 19:58:21,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31671.19 MB 2025-02-14 19:58:21,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40038.83 MB 2025-02-14 19:58:21,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 19:58:21,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28731.11 MB 2025-02-14 19:58:21,668 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 19:58:21,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:58:21,670 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:58:21,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:58:21,671 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:58:21,675 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:58:21,676 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:58:21,676 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:58:21,676 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 19:59:16,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:16,028 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:59:16,033 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:59:16,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:16,037 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:59:16,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:16,038 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:59:18,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:59:18,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:59:18,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.79 seconds 2025-02-14 19:59:18,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:18,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-14 19:59:18,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-14 19:59:18,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-14 19:59:18,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48406.46 MB 2025-02-14 19:59:18,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 19:59:18,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29349.64 MB 2025-02-14 19:59:18,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23701.31 MB 2025-02-14 19:59:18,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:59:18,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:59:18,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:59:18,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:18,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-14 19:59:18,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15061.45 MB 2025-02-14 19:59:18,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 190.95 MB 2025-02-14 19:59:18,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 19:59:18,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 19:59:18,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:59:18,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17192.59 MB 2025-02-14 19:59:19,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:59:19,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:59:19,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 19:59:19,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:19,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15061.45 MB 2025-02-14 19:59:19,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15279.09 MB 2025-02-14 19:59:19,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.65 MB 2025-02-14 19:59:19,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 19:59:19,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18584.96 MB 2025-02-14 19:59:19,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 19:59:19,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19232.13 MB 2025-02-14 19:59:19,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:59:19,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:59:19,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 19:59:19,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:19,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15279.02 MB 2025-02-14 19:59:19,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16053.55 MB 2025-02-14 19:59:19,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 774.52 MB 2025-02-14 19:59:19,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18584.96 MB 2025-02-14 19:59:19,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18584.96 MB 2025-02-14 19:59:19,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:59:19,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16634.70 MB 2025-02-14 19:59:19,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:59:19,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:59:19,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 19:59:19,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:19,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16053.55 MB 2025-02-14 19:59:19,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16972.74 MB 2025-02-14 19:59:19,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 919.20 MB 2025-02-14 19:59:19,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18584.96 MB 2025-02-14 19:59:19,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20331.89 MB 2025-02-14 19:59:19,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1746.93 MB 2025-02-14 19:59:19,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19250.06 MB 2025-02-14 19:59:19,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:59:19,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:59:19,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 19:59:19,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:19,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15279.02 MB 2025-02-14 19:59:19,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16972.74 MB 2025-02-14 19:59:19,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1693.72 MB 2025-02-14 19:59:19,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18584.96 MB 2025-02-14 19:59:19,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20331.89 MB 2025-02-14 19:59:19,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1746.93 MB 2025-02-14 19:59:19,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19250.06 MB 2025-02-14 19:59:19,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:59:19,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:59:19,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 19:59:19,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:19,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17601.50 MB 2025-02-14 19:59:19,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17915.97 MB 2025-02-14 19:59:19,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 314.47 MB 2025-02-14 19:59:19,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20331.89 MB 2025-02-14 19:59:19,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20501.76 MB 2025-02-14 19:59:19,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-14 19:59:19,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18213.24 MB 2025-02-14 19:59:19,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:59:19,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:59:19,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:59:19,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:19,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18085.26 MB 2025-02-14 19:59:19,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18298.88 MB 2025-02-14 19:59:19,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.62 MB 2025-02-14 19:59:19,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20501.76 MB 2025-02-14 19:59:19,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20501.76 MB 2025-02-14 19:59:19,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:59:19,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18325.47 MB 2025-02-14 19:59:19,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:59:19,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:59:19,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.78 seconds 2025-02-14 19:59:19,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:19,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-14 19:59:19,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18499.90 MB 2025-02-14 19:59:19,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4900.58 MB 2025-02-14 19:59:19,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48406.46 MB 2025-02-14 19:59:19,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20501.76 MB 2025-02-14 19:59:19,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27904.70 MB 2025-02-14 19:59:19,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18499.90 MB 2025-02-14 19:59:20,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:59:20,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:59:20,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 19:59:20,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:20,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18499.90 MB 2025-02-14 19:59:20,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17488.67 MB 2025-02-14 19:59:20,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1011.23 MB 2025-02-14 19:59:20,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20501.76 MB 2025-02-14 19:59:20,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20501.76 MB 2025-02-14 19:59:20,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:59:20,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19203.00 MB 2025-02-14 19:59:20,106 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 19:59:20,106 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:59:20,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:59:20,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:59:20,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:59:20,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:20,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17488.67 MB 2025-02-14 19:59:20,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25926.15 MB 2025-02-14 19:59:20,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 19:59:20,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20501.76 MB 2025-02-14 19:59:20,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30987.52 MB 2025-02-14 19:59:20,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 19:59:20,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25926.15 MB 2025-02-14 19:59:20,271 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 19:59:20,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:20,273 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:59:20,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:20,273 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:59:20,278 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:59:20,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:20,279 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:59:20,279 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:59:29,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:29,001 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 19:59:29,009 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 19:59:29,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:29,015 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 19:59:29,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:29,017 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 19:59:48,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 19:59:48,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 19:59:48,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.44 seconds 2025-02-14 19:59:48,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:48,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21727.68 MB 2025-02-14 19:59:48,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26176.14 MB 2025-02-14 19:59:48,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4448.45 MB 2025-02-14 19:59:48,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39376.13 MB 2025-02-14 19:59:48,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35146.17 MB 2025-02-14 19:59:48,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4229.96 MB 2025-02-14 19:59:48,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35049.43 MB 2025-02-14 19:59:48,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 19:59:48,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 19:59:48,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 19:59:48,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:48,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26176.14 MB 2025-02-14 19:59:48,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22312.60 MB 2025-02-14 19:59:48,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3863.53 MB 2025-02-14 19:59:48,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35146.17 MB 2025-02-14 19:59:48,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44096.82 MB 2025-02-14 19:59:48,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8950.64 MB 2025-02-14 19:59:48,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39485.88 MB 2025-02-14 19:59:50,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 19:59:50,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 19:59:50,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 19:59:50,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:50,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22312.60 MB 2025-02-14 19:59:50,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22843.44 MB 2025-02-14 19:59:50,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 19:59:50,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44096.82 MB 2025-02-14 19:59:50,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30696.01 MB 2025-02-14 19:59:50,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13400.80 MB 2025-02-14 19:59:50,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26821.99 MB 2025-02-14 19:59:50,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 19:59:50,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 19:59:50,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 19:59:50,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:50,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 19:59:50,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24732.98 MB 2025-02-14 19:59:50,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 19:59:50,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 19:59:50,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30696.01 MB 2025-02-14 19:59:50,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:59:50,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26150.41 MB 2025-02-14 19:59:50,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 19:59:50,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 19:59:50,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:59:50,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:50,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24732.98 MB 2025-02-14 19:59:50,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 19:59:50,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 19:59:50,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 19:59:50,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34470.89 MB 2025-02-14 19:59:50,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 19:59:50,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 19:59:50,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 19:59:50,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 19:59:50,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 19:59:50,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:50,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 19:59:50,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 19:59:50,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 19:59:50,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 19:59:50,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34470.89 MB 2025-02-14 19:59:50,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 19:59:50,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 19:59:50,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 19:59:50,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 19:59:50,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 19:59:50,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:50,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28508.38 MB 2025-02-14 19:59:50,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.38 MB 2025-02-14 19:59:50,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 19:59:50,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34470.89 MB 2025-02-14 19:59:50,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34888.22 MB 2025-02-14 19:59:50,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 19:59:50,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29983.17 MB 2025-02-14 19:59:50,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 19:59:50,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 19:59:50,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:59:50,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:50,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.27 MB 2025-02-14 19:59:50,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29916.61 MB 2025-02-14 19:59:50,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 19:59:50,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34888.22 MB 2025-02-14 19:59:50,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34888.22 MB 2025-02-14 19:59:50,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:59:50,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30148.39 MB 2025-02-14 19:59:50,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 19:59:50,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 19:59:50,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.92 seconds 2025-02-14 19:59:50,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:50,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17348.19 MB 2025-02-14 19:59:50,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30117.10 MB 2025-02-14 19:59:50,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12768.90 MB 2025-02-14 19:59:50,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39376.13 MB 2025-02-14 19:59:50,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34888.22 MB 2025-02-14 19:59:50,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4487.91 MB 2025-02-14 19:59:50,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30148.39 MB 2025-02-14 19:59:51,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 19:59:51,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 19:59:51,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 19:59:51,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:51,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30117.10 MB 2025-02-14 19:59:51,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22340.23 MB 2025-02-14 19:59:51,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7776.86 MB 2025-02-14 19:59:51,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34888.22 MB 2025-02-14 19:59:51,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34888.22 MB 2025-02-14 19:59:51,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 19:59:51,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32618.63 MB 2025-02-14 19:59:51,245 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 19:59:51,245 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 19:59:51,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 19:59:51,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 19:59:51,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 19:59:51,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 19:59:51,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22340.23 MB 2025-02-14 19:59:51,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30745.35 MB 2025-02-14 19:59:51,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 19:59:51,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34888.22 MB 2025-02-14 19:59:51,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43245.37 MB 2025-02-14 19:59:51,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 19:59:51,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30745.35 MB 2025-02-14 19:59:51,498 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 19:59:51,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:51,499 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 19:59:51,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:51,500 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 19:59:51,506 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 19:59:51,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 19:59:51,507 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 19:59:51,507 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 20:01:39,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:01:39,158 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:01:39,163 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:01:39,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:01:39,167 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 130, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:01:39,168 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:01:39,168 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 130, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:01:41,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:01:41,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:01:41,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.03 seconds 2025-02-14 20:01:41,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:41,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13874.57 MB 2025-02-14 20:01:41,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14334.63 MB 2025-02-14 20:01:41,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 460.06 MB 2025-02-14 20:01:41,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55780.05 MB 2025-02-14 20:01:41,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:01:41,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35777.41 MB 2025-02-14 20:01:41,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23345.94 MB 2025-02-14 20:01:41,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:01:41,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:01:41,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:01:41,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:41,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14334.63 MB 2025-02-14 20:01:41,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14557.53 MB 2025-02-14 20:01:41,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.90 MB 2025-02-14 20:01:41,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:01:41,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:01:41,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:01:41,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16167.76 MB 2025-02-14 20:01:41,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:01:41,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:01:41,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.62 seconds 2025-02-14 20:01:41,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:41,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14557.53 MB 2025-02-14 20:01:41,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14730.05 MB 2025-02-14 20:01:41,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-14 20:01:41,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:01:41,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:01:41,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:01:41,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18727.18 MB 2025-02-14 20:01:41,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:01:41,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:01:41,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:01:41,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:41,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14729.99 MB 2025-02-14 20:01:41,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15343.94 MB 2025-02-14 20:01:41,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-14 20:01:41,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:01:41,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:01:41,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:01:41,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15804.61 MB 2025-02-14 20:01:42,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:01:42,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:01:42,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:01:42,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:42,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15343.94 MB 2025-02-14 20:01:42,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16072.58 MB 2025-02-14 20:01:42,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-14 20:01:42,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:01:42,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:01:42,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:01:42,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.43 MB 2025-02-14 20:01:42,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:01:42,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:01:42,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:01:42,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:42,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14729.99 MB 2025-02-14 20:01:42,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16072.58 MB 2025-02-14 20:01:42,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-14 20:01:42,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:01:42,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:01:42,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:01:42,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.43 MB 2025-02-14 20:01:42,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:01:42,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:01:42,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:01:42,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:42,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16570.98 MB 2025-02-14 20:01:42,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16820.26 MB 2025-02-14 20:01:42,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-14 20:01:42,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:01:42,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20134.76 MB 2025-02-14 20:01:42,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 20:01:42,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17061.56 MB 2025-02-14 20:01:42,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:01:42,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:01:42,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:01:42,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:42,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16954.46 MB 2025-02-14 20:01:42,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17150.95 MB 2025-02-14 20:01:42,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.50 MB 2025-02-14 20:01:42,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20134.76 MB 2025-02-14 20:01:42,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20138.95 MB 2025-02-14 20:01:42,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 20:01:42,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17150.95 MB 2025-02-14 20:01:42,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:01:42,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:01:42,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.97 seconds 2025-02-14 20:01:42,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:42,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13421.64 MB 2025-02-14 20:01:42,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17343.22 MB 2025-02-14 20:01:42,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3921.59 MB 2025-02-14 20:01:42,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55780.05 MB 2025-02-14 20:01:42,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20138.95 MB 2025-02-14 20:01:42,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35641.10 MB 2025-02-14 20:01:42,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17343.22 MB 2025-02-14 20:01:42,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:01:42,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:01:42,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 20:01:42,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:42,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.38 MB 2025-02-14 20:01:42,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17015.44 MB 2025-02-14 20:01:42,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2882.06 MB 2025-02-14 20:01:42,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20138.95 MB 2025-02-14 20:01:42,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20138.95 MB 2025-02-14 20:01:42,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:01:42,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17303.61 MB 2025-02-14 20:01:42,422 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7804, cut from 7806 2025-02-14 20:01:42,423 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:01:42,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:01:42,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:01:42,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:01:42,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:01:42,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17015.44 MB 2025-02-14 20:01:42,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25085.63 MB 2025-02-14 20:01:42,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8070.20 MB 2025-02-14 20:01:42,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20138.95 MB 2025-02-14 20:01:42,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28162.65 MB 2025-02-14 20:01:42,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-14 20:01:42,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25085.63 MB 2025-02-14 20:01:42,580 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7596] 2025-02-14 20:01:42,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:01:42,582 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:01:42,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:01:42,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:01:42,587 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:01:42,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:01:42,589 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:01:42,589 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:02:16,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:02:16,130 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:02:16,135 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:02:16,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:02:16,138 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3208, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:02:16,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:02:16,139 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3208, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:03:05,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:03:05,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:03:05,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 49.33 seconds 2025-02-14 20:03:05,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:05,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35324.35 MB 2025-02-14 20:03:05,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46678.33 MB 2025-02-14 20:03:05,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11353.98 MB 2025-02-14 20:03:05,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58542.00 MB 2025-02-14 20:03:05,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51822.72 MB 2025-02-14 20:03:05,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6719.28 MB 2025-02-14 20:03:05,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58031.26 MB 2025-02-14 20:03:05,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:03:05,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:03:05,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 20:03:05,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:05,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46678.33 MB 2025-02-14 20:03:05,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32456.50 MB 2025-02-14 20:03:05,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14221.83 MB 2025-02-14 20:03:05,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51822.72 MB 2025-02-14 20:03:05,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57510.20 MB 2025-02-14 20:03:05,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5687.48 MB 2025-02-14 20:03:05,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55201.42 MB 2025-02-14 20:03:07,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:03:07,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:03:07,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 20:03:07,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:07,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32456.50 MB 2025-02-14 20:03:07,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32987.34 MB 2025-02-14 20:03:07,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:03:07,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57510.20 MB 2025-02-14 20:03:07,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36207.33 MB 2025-02-14 20:03:07,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21302.87 MB 2025-02-14 20:03:07,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36966.93 MB 2025-02-14 20:03:07,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:03:07,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:03:07,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:03:07,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:07,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32987.34 MB 2025-02-14 20:03:07,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34876.88 MB 2025-02-14 20:03:07,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:03:07,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36207.33 MB 2025-02-14 20:03:07,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38094.77 MB 2025-02-14 20:03:07,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:03:07,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36294.30 MB 2025-02-14 20:03:07,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:03:07,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:03:07,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:03:07,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:07,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34876.88 MB 2025-02-14 20:03:07,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37118.73 MB 2025-02-14 20:03:07,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:03:07,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38094.77 MB 2025-02-14 20:03:07,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44228.94 MB 2025-02-14 20:03:07,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:03:07,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42663.01 MB 2025-02-14 20:03:07,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:03:07,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:03:07,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:03:07,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:07,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32987.34 MB 2025-02-14 20:03:07,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37118.73 MB 2025-02-14 20:03:07,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:03:07,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36207.33 MB 2025-02-14 20:03:07,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44228.94 MB 2025-02-14 20:03:07,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 20:03:07,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42663.01 MB 2025-02-14 20:03:07,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:03:07,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:03:07,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 20:03:07,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:07,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38652.27 MB 2025-02-14 20:03:07,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39419.28 MB 2025-02-14 20:03:07,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:03:07,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44228.94 MB 2025-02-14 20:03:07,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44646.27 MB 2025-02-14 20:03:07,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:03:07,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40127.06 MB 2025-02-14 20:03:07,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:03:07,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:03:07,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:03:07,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:07,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39832.16 MB 2025-02-14 20:03:07,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40058.98 MB 2025-02-14 20:03:07,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.81 MB 2025-02-14 20:03:07,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44646.27 MB 2025-02-14 20:03:07,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44646.27 MB 2025-02-14 20:03:07,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:03:07,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40253.68 MB 2025-02-14 20:03:07,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:03:07,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:03:07,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.82 seconds 2025-02-14 20:03:07,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:07,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24146.53 MB 2025-02-14 20:03:07,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40260.05 MB 2025-02-14 20:03:07,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16113.52 MB 2025-02-14 20:03:07,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47364.18 MB 2025-02-14 20:03:07,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44646.27 MB 2025-02-14 20:03:07,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2717.91 MB 2025-02-14 20:03:07,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40260.05 MB 2025-02-14 20:03:08,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:03:08,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:03:08,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:03:08,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:08,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40260.05 MB 2025-02-14 20:03:08,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29150.92 MB 2025-02-14 20:03:08,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11109.13 MB 2025-02-14 20:03:08,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44646.27 MB 2025-02-14 20:03:08,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44646.27 MB 2025-02-14 20:03:08,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:03:08,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41264.72 MB 2025-02-14 20:03:08,251 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:03:08,251 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:03:08,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:03:08,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:03:08,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:03:08,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:03:08,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29150.92 MB 2025-02-14 20:03:08,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37589.94 MB 2025-02-14 20:03:08,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:03:08,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44646.27 MB 2025-02-14 20:03:08,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48840.57 MB 2025-02-14 20:03:08,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 20:03:08,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37589.94 MB 2025-02-14 20:03:08,424 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:03:08,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:03:08,425 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:03:08,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:03:08,426 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:03:08,432 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:03:08,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:03:08,433 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:03:08,433 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:03:51,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:03:51,308 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:03:51,315 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:03:51,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:03:51,321 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:03:51,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:03:51,323 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:04:03,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:04:03,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:04:03,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.54 seconds 2025-02-14 20:04:03,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:03,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.98 MB 2025-02-14 20:04:03,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21459.49 MB 2025-02-14 20:04:03,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2860.52 MB 2025-02-14 20:04:03,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61425.58 MB 2025-02-14 20:04:03,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26585.60 MB 2025-02-14 20:04:03,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34839.99 MB 2025-02-14 20:04:03,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30335.27 MB 2025-02-14 20:04:03,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:04:03,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:04:03,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 20:04:03,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:03,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21459.49 MB 2025-02-14 20:04:03,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19979.44 MB 2025-02-14 20:04:03,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1480.06 MB 2025-02-14 20:04:03,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26585.60 MB 2025-02-14 20:04:03,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-14 20:04:03,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8537.51 MB 2025-02-14 20:04:03,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31347.38 MB 2025-02-14 20:04:05,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:04:05,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:04:05,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:04:05,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:05,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19979.44 MB 2025-02-14 20:04:05,848 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20510.28 MB 2025-02-14 20:04:05,848 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:04:05,848 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35123.10 MB 2025-02-14 20:04:05,848 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25849.50 MB 2025-02-14 20:04:05,848 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9273.61 MB 2025-02-14 20:04:05,848 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24488.83 MB 2025-02-14 20:04:05,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:04:05,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:04:05,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:04:05,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:05,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20510.28 MB 2025-02-14 20:04:05,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22399.81 MB 2025-02-14 20:04:05,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:04:05,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25849.50 MB 2025-02-14 20:04:05,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25849.50 MB 2025-02-14 20:04:05,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:04:05,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23817.24 MB 2025-02-14 20:04:06,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:04:06,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:04:06,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:04:06,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:06,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22399.81 MB 2025-02-14 20:04:06,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24641.67 MB 2025-02-14 20:04:06,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:04:06,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25849.50 MB 2025-02-14 20:04:06,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31983.67 MB 2025-02-14 20:04:06,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:04:06,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30185.95 MB 2025-02-14 20:04:06,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:04:06,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:04:06,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:04:06,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:06,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20510.28 MB 2025-02-14 20:04:06,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24641.67 MB 2025-02-14 20:04:06,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:04:06,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25849.50 MB 2025-02-14 20:04:06,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31983.67 MB 2025-02-14 20:04:06,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:04:06,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30185.95 MB 2025-02-14 20:04:06,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:04:06,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:04:06,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:04:06,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:06,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26175.21 MB 2025-02-14 20:04:06,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26942.21 MB 2025-02-14 20:04:06,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:04:06,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31983.67 MB 2025-02-14 20:04:06,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32398.90 MB 2025-02-14 20:04:06,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:04:06,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27650.00 MB 2025-02-14 20:04:06,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:04:06,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:04:06,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:04:06,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:06,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27355.10 MB 2025-02-14 20:04:06,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27584.00 MB 2025-02-14 20:04:06,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.90 MB 2025-02-14 20:04:06,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32398.90 MB 2025-02-14 20:04:06,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32398.90 MB 2025-02-14 20:04:06,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:04:06,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27799.52 MB 2025-02-14 20:04:06,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:04:06,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:04:06,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.93 seconds 2025-02-14 20:04:06,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:06,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15783.84 MB 2025-02-14 20:04:06,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27784.49 MB 2025-02-14 20:04:06,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12000.64 MB 2025-02-14 20:04:06,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61425.58 MB 2025-02-14 20:04:06,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32398.90 MB 2025-02-14 20:04:06,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29026.68 MB 2025-02-14 20:04:06,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27799.52 MB 2025-02-14 20:04:06,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:04:06,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:04:06,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:04:06,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:06,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27784.49 MB 2025-02-14 20:04:06,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20777.66 MB 2025-02-14 20:04:06,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7006.82 MB 2025-02-14 20:04:06,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32398.90 MB 2025-02-14 20:04:06,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32398.90 MB 2025-02-14 20:04:06,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:04:06,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30287.55 MB 2025-02-14 20:04:06,546 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 20:04:06,546 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 20:04:06,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:04:06,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:04:06,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:04:06,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:06,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20777.66 MB 2025-02-14 20:04:06,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29187.47 MB 2025-02-14 20:04:06,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 20:04:06,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32398.90 MB 2025-02-14 20:04:06,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40760.25 MB 2025-02-14 20:04:06,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 20:04:06,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29187.47 MB 2025-02-14 20:04:06,716 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 20:04:06,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:06,717 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:04:06,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:06,718 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:04:06,723 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:04:06,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:06,724 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:04:06,724 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 20:04:18,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:18,236 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:04:18,241 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:04:18,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:18,244 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1025, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:04:18,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:18,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1025, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:04:34,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:04:34,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:04:34,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.97 seconds 2025-02-14 20:04:34,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:34,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20111.07 MB 2025-02-14 20:04:34,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23739.14 MB 2025-02-14 20:04:34,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3628.07 MB 2025-02-14 20:04:34,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53301.22 MB 2025-02-14 20:04:34,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30104.62 MB 2025-02-14 20:04:34,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23196.60 MB 2025-02-14 20:04:34,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32753.33 MB 2025-02-14 20:04:34,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:04:34,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:04:34,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:04:34,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:34,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23739.14 MB 2025-02-14 20:04:34,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21106.51 MB 2025-02-14 20:04:34,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2632.64 MB 2025-02-14 20:04:34,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30104.62 MB 2025-02-14 20:04:34,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39736.84 MB 2025-02-14 20:04:34,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9632.22 MB 2025-02-14 20:04:34,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35117.68 MB 2025-02-14 20:04:36,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:04:36,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:04:36,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:04:36,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21106.51 MB 2025-02-14 20:04:36,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21637.35 MB 2025-02-14 20:04:36,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:04:36,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39736.84 MB 2025-02-14 20:04:36,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27892.12 MB 2025-02-14 20:04:36,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11844.71 MB 2025-02-14 20:04:36,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25615.89 MB 2025-02-14 20:04:36,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:04:36,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:04:36,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:04:36,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21637.35 MB 2025-02-14 20:04:36,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23526.88 MB 2025-02-14 20:04:36,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:04:36,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27892.12 MB 2025-02-14 20:04:36,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27892.12 MB 2025-02-14 20:04:36,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:04:36,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24944.31 MB 2025-02-14 20:04:36,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:04:36,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:04:36,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:04:36,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23526.88 MB 2025-02-14 20:04:36,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25768.74 MB 2025-02-14 20:04:36,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:04:36,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27892.12 MB 2025-02-14 20:04:36,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 20:04:36,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 20:04:36,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31313.02 MB 2025-02-14 20:04:36,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:04:36,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:04:36,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:04:36,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21637.35 MB 2025-02-14 20:04:36,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25768.74 MB 2025-02-14 20:04:36,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:04:36,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27892.12 MB 2025-02-14 20:04:36,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 20:04:36,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 20:04:36,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31313.02 MB 2025-02-14 20:04:36,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:04:36,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:04:36,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 20:04:36,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27302.28 MB 2025-02-14 20:04:36,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28069.28 MB 2025-02-14 20:04:36,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:04:36,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 20:04:36,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33495.71 MB 2025-02-14 20:04:36,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 20:04:36,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28777.07 MB 2025-02-14 20:04:36,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:04:36,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:04:36,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:04:36,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28482.17 MB 2025-02-14 20:04:36,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28709.26 MB 2025-02-14 20:04:36,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.09 MB 2025-02-14 20:04:36,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33495.71 MB 2025-02-14 20:04:36,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33495.71 MB 2025-02-14 20:04:36,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:04:36,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28941.50 MB 2025-02-14 20:04:36,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:04:36,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:04:36,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.38 seconds 2025-02-14 20:04:36,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16539.89 MB 2025-02-14 20:04:36,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28909.43 MB 2025-02-14 20:04:36,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12369.54 MB 2025-02-14 20:04:36,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53301.22 MB 2025-02-14 20:04:36,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33495.71 MB 2025-02-14 20:04:36,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19805.50 MB 2025-02-14 20:04:36,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28941.50 MB 2025-02-14 20:04:36,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:04:36,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:04:36,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:04:36,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28909.43 MB 2025-02-14 20:04:36,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21530.50 MB 2025-02-14 20:04:36,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7378.92 MB 2025-02-14 20:04:36,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33495.71 MB 2025-02-14 20:04:36,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33495.71 MB 2025-02-14 20:04:36,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:04:36,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31410.05 MB 2025-02-14 20:04:36,915 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 20:04:36,915 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:04:36,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:04:36,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:04:36,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:04:36,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:04:36,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21530.50 MB 2025-02-14 20:04:36,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29931.44 MB 2025-02-14 20:04:36,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 20:04:36,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33495.71 MB 2025-02-14 20:04:36,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41848.67 MB 2025-02-14 20:04:36,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 20:04:36,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29931.44 MB 2025-02-14 20:04:37,082 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 20:04:37,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:37,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:04:37,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:37,085 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:04:37,089 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:04:37,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:04:37,091 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:04:37,091 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:05:09,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:09,520 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:05:09,524 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:05:09,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:09,528 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:05:09,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:09,529 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:05:12,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:05:12,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:05:12,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.25 seconds 2025-02-14 20:05:12,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:12,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14425.05 MB 2025-02-14 20:05:12,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15164.69 MB 2025-02-14 20:05:12,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 739.64 MB 2025-02-14 20:05:12,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54377.05 MB 2025-02-14 20:05:12,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 20:05:12,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33902.56 MB 2025-02-14 20:05:12,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24122.92 MB 2025-02-14 20:05:12,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:05:12,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:05:12,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:05:12,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:12,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15164.69 MB 2025-02-14 20:05:12,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15522.98 MB 2025-02-14 20:05:12,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 358.29 MB 2025-02-14 20:05:12,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 20:05:12,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 20:05:12,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:12,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18104.23 MB 2025-02-14 20:05:13,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:05:13,809 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:05:13,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-14 20:05:13,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:13,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15522.98 MB 2025-02-14 20:05:13,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15800.34 MB 2025-02-14 20:05:13,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-14 20:05:13,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 20:05:13,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:05:13,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 20:05:13,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.16 MB 2025-02-14 20:05:13,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:05:13,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:05:13,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:05:13,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:13,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15800.34 MB 2025-02-14 20:05:13,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16787.39 MB 2025-02-14 20:05:13,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-14 20:05:13,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:05:13,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 20:05:13,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:13,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17528.00 MB 2025-02-14 20:05:13,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:05:13,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:05:13,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 20:05:13,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:13,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16787.39 MB 2025-02-14 20:05:13,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17958.79 MB 2025-02-14 20:05:13,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.40 MB 2025-02-14 20:05:13,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:05:13,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22477.28 MB 2025-02-14 20:05:13,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2474.64 MB 2025-02-14 20:05:13,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20857.48 MB 2025-02-14 20:05:13,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:05:13,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:05:13,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:05:13,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:13,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15800.34 MB 2025-02-14 20:05:13,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17958.79 MB 2025-02-14 20:05:13,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.44 MB 2025-02-14 20:05:13,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 20:05:13,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22477.28 MB 2025-02-14 20:05:13,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2474.64 MB 2025-02-14 20:05:13,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20857.48 MB 2025-02-14 20:05:14,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:05:14,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:05:14,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:05:14,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:14,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18760.06 MB 2025-02-14 20:05:14,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19162.66 MB 2025-02-14 20:05:14,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.59 MB 2025-02-14 20:05:14,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22477.28 MB 2025-02-14 20:05:14,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22693.28 MB 2025-02-14 20:05:14,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-14 20:05:14,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19533.14 MB 2025-02-14 20:05:14,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:05:14,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:05:14,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:05:14,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:14,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19378.40 MB 2025-02-14 20:05:14,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19607.30 MB 2025-02-14 20:05:14,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.90 MB 2025-02-14 20:05:14,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22693.28 MB 2025-02-14 20:05:14,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22693.28 MB 2025-02-14 20:05:14,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:14,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19673.72 MB 2025-02-14 20:05:14,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:05:14,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:05:14,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.50 seconds 2025-02-14 20:05:14,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:14,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13696.88 MB 2025-02-14 20:05:14,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19808.37 MB 2025-02-14 20:05:14,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6111.49 MB 2025-02-14 20:05:14,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54377.05 MB 2025-02-14 20:05:14,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22693.28 MB 2025-02-14 20:05:14,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31683.77 MB 2025-02-14 20:05:14,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19808.37 MB 2025-02-14 20:05:14,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:05:14,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:05:14,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:05:14,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:14,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14787.68 MB 2025-02-14 20:05:14,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17801.72 MB 2025-02-14 20:05:14,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 20:05:14,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22693.28 MB 2025-02-14 20:05:14,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22693.28 MB 2025-02-14 20:05:14,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:14,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18103.09 MB 2025-02-14 20:05:14,315 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:05:14,315 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:05:14,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:05:14,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:05:14,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:05:14,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:14,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17801.72 MB 2025-02-14 20:05:14,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26240.74 MB 2025-02-14 20:05:14,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:05:14,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22693.28 MB 2025-02-14 20:05:14,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31083.99 MB 2025-02-14 20:05:14,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:05:14,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26240.74 MB 2025-02-14 20:05:14,477 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:05:14,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:14,479 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:05:14,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:14,480 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:05:14,484 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:05:14,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:14,485 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:05:14,485 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:05:27,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:27,651 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:05:27,656 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:05:27,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:27,660 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 716, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:05:27,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:27,661 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 716, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:05:38,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:05:38,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:05:38,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.12 seconds 2025-02-14 20:05:38,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:38,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17957.91 MB 2025-02-14 20:05:38,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20491.79 MB 2025-02-14 20:05:38,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2533.88 MB 2025-02-14 20:05:38,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43669.00 MB 2025-02-14 20:05:38,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22307.41 MB 2025-02-14 20:05:38,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21361.59 MB 2025-02-14 20:05:38,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29467.71 MB 2025-02-14 20:05:38,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:05:38,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:05:38,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 20:05:38,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:38,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20491.79 MB 2025-02-14 20:05:38,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19501.16 MB 2025-02-14 20:05:38,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -990.63 MB 2025-02-14 20:05:38,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22307.41 MB 2025-02-14 20:05:38,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35414.61 MB 2025-02-14 20:05:38,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13107.20 MB 2025-02-14 20:05:38,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29594.03 MB 2025-02-14 20:05:40,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:05:40,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:05:40,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:05:40,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:40,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19501.16 MB 2025-02-14 20:05:40,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20032.00 MB 2025-02-14 20:05:40,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:05:40,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35414.61 MB 2025-02-14 20:05:40,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25140.66 MB 2025-02-14 20:05:40,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10273.95 MB 2025-02-14 20:05:40,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24010.55 MB 2025-02-14 20:05:40,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:05:40,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:05:40,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:05:40,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:40,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20032.00 MB 2025-02-14 20:05:40,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21921.54 MB 2025-02-14 20:05:40,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:05:40,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25140.66 MB 2025-02-14 20:05:40,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25140.66 MB 2025-02-14 20:05:40,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:40,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23338.96 MB 2025-02-14 20:05:40,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:05:40,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:05:40,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:05:40,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:40,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21921.54 MB 2025-02-14 20:05:40,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24164.44 MB 2025-02-14 20:05:40,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-14 20:05:40,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25140.66 MB 2025-02-14 20:05:40,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31511.81 MB 2025-02-14 20:05:40,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-14 20:05:40,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29708.72 MB 2025-02-14 20:05:40,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:05:40,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:05:40,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:05:40,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:40,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20032.00 MB 2025-02-14 20:05:40,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24164.44 MB 2025-02-14 20:05:40,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-14 20:05:40,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25140.66 MB 2025-02-14 20:05:40,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31511.81 MB 2025-02-14 20:05:40,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-14 20:05:40,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29708.72 MB 2025-02-14 20:05:41,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:05:41,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:05:41,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 20:05:41,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:41,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25697.98 MB 2025-02-14 20:05:41,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26464.98 MB 2025-02-14 20:05:41,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:05:41,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31511.81 MB 2025-02-14 20:05:41,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31929.14 MB 2025-02-14 20:05:41,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:05:41,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27172.77 MB 2025-02-14 20:05:41,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:05:41,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:05:41,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:05:41,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:41,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26877.87 MB 2025-02-14 20:05:41,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27107.46 MB 2025-02-14 20:05:41,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.59 MB 2025-02-14 20:05:41,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31929.14 MB 2025-02-14 20:05:41,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31929.14 MB 2025-02-14 20:05:41,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:41,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27302.97 MB 2025-02-14 20:05:41,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:05:41,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:05:41,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.55 seconds 2025-02-14 20:05:41,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:41,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15463.31 MB 2025-02-14 20:05:41,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27308.53 MB 2025-02-14 20:05:41,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11845.23 MB 2025-02-14 20:05:41,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43669.00 MB 2025-02-14 20:05:41,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31929.14 MB 2025-02-14 20:05:41,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11739.86 MB 2025-02-14 20:05:41,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27308.53 MB 2025-02-14 20:05:41,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:05:41,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:05:41,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:05:41,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:41,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27308.53 MB 2025-02-14 20:05:41,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20467.70 MB 2025-02-14 20:05:41,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6840.84 MB 2025-02-14 20:05:41,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31929.14 MB 2025-02-14 20:05:41,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31929.14 MB 2025-02-14 20:05:41,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:41,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29820.20 MB 2025-02-14 20:05:41,498 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:05:41,499 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:05:41,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:05:41,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:05:41,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:05:41,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:41,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20467.70 MB 2025-02-14 20:05:41,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28906.72 MB 2025-02-14 20:05:41,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:05:41,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31929.14 MB 2025-02-14 20:05:41,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36125.54 MB 2025-02-14 20:05:41,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 20:05:41,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28906.72 MB 2025-02-14 20:05:41,660 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:05:41,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:41,662 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:05:41,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:41,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:05:41,667 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:05:41,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:41,669 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:05:41,669 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:05:51,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:51,165 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:05:51,170 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:05:51,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:51,174 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:05:51,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:51,175 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:05:55,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:05:55,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:05:55,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.68 seconds 2025-02-14 20:05:55,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:55,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15066.12 MB 2025-02-14 20:05:55,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16131.34 MB 2025-02-14 20:05:55,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1065.22 MB 2025-02-14 20:05:55,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48710.55 MB 2025-02-14 20:05:55,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23725.08 MB 2025-02-14 20:05:55,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24985.47 MB 2025-02-14 20:05:55,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24990.48 MB 2025-02-14 20:05:55,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:05:55,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:05:55,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:05:55,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:55,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16131.34 MB 2025-02-14 20:05:55,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16647.38 MB 2025-02-14 20:05:55,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 516.03 MB 2025-02-14 20:05:55,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23725.08 MB 2025-02-14 20:05:55,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23725.08 MB 2025-02-14 20:05:55,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:55,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20405.21 MB 2025-02-14 20:05:57,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:05:57,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:05:57,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.45 seconds 2025-02-14 20:05:57,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16647.38 MB 2025-02-14 20:05:57,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17046.83 MB 2025-02-14 20:05:57,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 399.46 MB 2025-02-14 20:05:57,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23725.08 MB 2025-02-14 20:05:57,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20478.69 MB 2025-02-14 20:05:57,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3246.39 MB 2025-02-14 20:05:57,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20987.93 MB 2025-02-14 20:05:57,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:05:57,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:05:57,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:05:57,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17046.83 MB 2025-02-14 20:05:57,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18470.80 MB 2025-02-14 20:05:57,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1423.97 MB 2025-02-14 20:05:57,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20478.69 MB 2025-02-14 20:05:57,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21900.56 MB 2025-02-14 20:05:57,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1421.87 MB 2025-02-14 20:05:57,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19537.42 MB 2025-02-14 20:05:57,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:05:57,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:05:57,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:05:57,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18470.80 MB 2025-02-14 20:05:57,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20157.81 MB 2025-02-14 20:05:57,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1687.01 MB 2025-02-14 20:05:57,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21900.56 MB 2025-02-14 20:05:57,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25811.75 MB 2025-02-14 20:05:57,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3911.19 MB 2025-02-14 20:05:57,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24333.02 MB 2025-02-14 20:05:57,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:05:57,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:05:57,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 20:05:57,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17046.83 MB 2025-02-14 20:05:57,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20157.81 MB 2025-02-14 20:05:57,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3110.98 MB 2025-02-14 20:05:57,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20478.69 MB 2025-02-14 20:05:57,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25811.75 MB 2025-02-14 20:05:57,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5333.06 MB 2025-02-14 20:05:57,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24333.02 MB 2025-02-14 20:05:57,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:05:57,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:05:57,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:05:57,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21311.80 MB 2025-02-14 20:05:57,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21888.97 MB 2025-02-14 20:05:57,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 577.17 MB 2025-02-14 20:05:57,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25811.75 MB 2025-02-14 20:05:57,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26120.03 MB 2025-02-14 20:05:57,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 308.28 MB 2025-02-14 20:05:57,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22421.58 MB 2025-02-14 20:05:57,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:05:57,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:05:57,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:05:57,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22199.68 MB 2025-02-14 20:05:57,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22416.99 MB 2025-02-14 20:05:57,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.32 MB 2025-02-14 20:05:57,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26120.03 MB 2025-02-14 20:05:57,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26120.03 MB 2025-02-14 20:05:57,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:05:57,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22549.67 MB 2025-02-14 20:05:57,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:05:57,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:05:57,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.46 seconds 2025-02-14 20:05:57,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14017.41 MB 2025-02-14 20:05:57,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22618.07 MB 2025-02-14 20:05:57,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8600.65 MB 2025-02-14 20:05:57,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48710.55 MB 2025-02-14 20:05:57,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26120.03 MB 2025-02-14 20:05:57,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22590.52 MB 2025-02-14 20:05:57,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22618.07 MB 2025-02-14 20:05:57,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:05:57,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:05:57,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:05:57,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22618.07 MB 2025-02-14 20:05:57,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25632.10 MB 2025-02-14 20:05:57,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 20:05:57,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26120.03 MB 2025-02-14 20:05:57,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26925.33 MB 2025-02-14 20:05:57,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 805.31 MB 2025-02-14 20:05:57,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25933.73 MB 2025-02-14 20:05:57,930 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:05:57,930 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:05:57,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:05:57,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:05:57,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 20:05:57,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:05:57,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18554.59 MB 2025-02-14 20:05:57,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26993.62 MB 2025-02-14 20:05:57,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:05:57,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26925.33 MB 2025-02-14 20:05:57,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35316.04 MB 2025-02-14 20:05:57,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:05:57,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.62 MB 2025-02-14 20:05:58,108 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:05:58,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:58,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:05:58,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:58,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:05:58,114 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:05:58,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:05:58,116 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:05:58,116 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:06:16,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:16,591 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:06:16,599 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:06:16,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:16,606 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 176, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:06:16,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:16,608 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 176, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:06:19,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:06:19,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:06:19,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.81 seconds 2025-02-14 20:06:19,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:19,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14195.10 MB 2025-02-14 20:06:19,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14817.96 MB 2025-02-14 20:06:19,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 622.85 MB 2025-02-14 20:06:19,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47901.05 MB 2025-02-14 20:06:19,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22351.45 MB 2025-02-14 20:06:19,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25549.60 MB 2025-02-14 20:06:19,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23666.47 MB 2025-02-14 20:06:19,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:06:19,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:06:19,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:06:19,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:19,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14817.96 MB 2025-02-14 20:06:19,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15119.73 MB 2025-02-14 20:06:19,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 301.77 MB 2025-02-14 20:06:19,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22351.45 MB 2025-02-14 20:06:19,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22351.45 MB 2025-02-14 20:06:19,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:06:19,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.75 MB 2025-02-14 20:06:20,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:06:20,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:06:20,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 20:06:20,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15119.73 MB 2025-02-14 20:06:20,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15353.30 MB 2025-02-14 20:06:20,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.57 MB 2025-02-14 20:06:20,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22351.45 MB 2025-02-14 20:06:20,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19895.68 MB 2025-02-14 20:06:20,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2455.76 MB 2025-02-14 20:06:20,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19290.42 MB 2025-02-14 20:06:20,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:06:20,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:06:20,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:06:20,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15353.23 MB 2025-02-14 20:06:20,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16184.43 MB 2025-02-14 20:06:20,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.19 MB 2025-02-14 20:06:20,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19895.68 MB 2025-02-14 20:06:20,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19895.68 MB 2025-02-14 20:06:20,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:06:20,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16808.10 MB 2025-02-14 20:06:20,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:06:20,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:06:20,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:06:20,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16184.43 MB 2025-02-14 20:06:20,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17170.88 MB 2025-02-14 20:06:20,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 986.45 MB 2025-02-14 20:06:20,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19895.68 MB 2025-02-14 20:06:20,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20933.77 MB 2025-02-14 20:06:20,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1038.09 MB 2025-02-14 20:06:20,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19610.33 MB 2025-02-14 20:06:20,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:06:20,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:06:20,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 20:06:20,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15353.23 MB 2025-02-14 20:06:20,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17170.88 MB 2025-02-14 20:06:20,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1817.65 MB 2025-02-14 20:06:20,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19895.68 MB 2025-02-14 20:06:20,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20933.77 MB 2025-02-14 20:06:20,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1038.09 MB 2025-02-14 20:06:20,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19610.33 MB 2025-02-14 20:06:20,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:06:20,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:06:20,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:06:20,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17845.64 MB 2025-02-14 20:06:20,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18183.12 MB 2025-02-14 20:06:20,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.48 MB 2025-02-14 20:06:20,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20933.77 MB 2025-02-14 20:06:20,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21112.03 MB 2025-02-14 20:06:20,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 178.26 MB 2025-02-14 20:06:20,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18500.16 MB 2025-02-14 20:06:20,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:06:20,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:06:20,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:06:20,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18364.80 MB 2025-02-14 20:06:20,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18593.78 MB 2025-02-14 20:06:20,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.98 MB 2025-02-14 20:06:20,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21112.03 MB 2025-02-14 20:06:20,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21112.03 MB 2025-02-14 20:06:20,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:06:20,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18632.60 MB 2025-02-14 20:06:20,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:06:20,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:06:20,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.98 seconds 2025-02-14 20:06:20,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13581.90 MB 2025-02-14 20:06:20,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18794.70 MB 2025-02-14 20:06:20,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5212.80 MB 2025-02-14 20:06:20,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47901.05 MB 2025-02-14 20:06:20,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21112.03 MB 2025-02-14 20:06:20,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26789.02 MB 2025-02-14 20:06:20,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18794.70 MB 2025-02-14 20:06:20,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:06:20,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:06:20,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 20:06:20,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18794.70 MB 2025-02-14 20:06:20,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17527.77 MB 2025-02-14 20:06:20,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1266.93 MB 2025-02-14 20:06:20,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21112.03 MB 2025-02-14 20:06:20,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21112.03 MB 2025-02-14 20:06:20,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:06:20,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19029.92 MB 2025-02-14 20:06:20,903 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 20:06:20,903 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:06:20,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:06:20,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:06:20,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:06:20,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:06:20,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17527.77 MB 2025-02-14 20:06:20,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25961.07 MB 2025-02-14 20:06:20,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 20:06:20,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21112.03 MB 2025-02-14 20:06:20,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29496.44 MB 2025-02-14 20:06:20,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 20:06:20,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25961.07 MB 2025-02-14 20:06:21,160 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 20:06:21,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:21,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:06:21,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:21,165 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:06:21,172 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:06:21,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:21,174 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:06:21,175 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:06:56,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:56,841 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:06:56,846 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:06:56,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:56,849 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:06:56,850 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:06:56,850 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:07:02,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:07:02,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:07:02,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.85 seconds 2025-02-14 20:07:02,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:02,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-14 20:07:02,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16950.90 MB 2025-02-14 20:07:02,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1341.26 MB 2025-02-14 20:07:02,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37880.86 MB 2025-02-14 20:07:02,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20336.08 MB 2025-02-14 20:07:02,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17544.77 MB 2025-02-14 20:07:02,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25760.49 MB 2025-02-14 20:07:02,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:07:02,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:07:02,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:07:02,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:02,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16950.90 MB 2025-02-14 20:07:02,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17601.59 MB 2025-02-14 20:07:02,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 650.69 MB 2025-02-14 20:07:02,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20336.08 MB 2025-02-14 20:07:02,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24341.64 MB 2025-02-14 20:07:02,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4005.56 MB 2025-02-14 20:07:02,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22315.85 MB 2025-02-14 20:07:04,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:07:04,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:07:04,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.82 seconds 2025-02-14 20:07:04,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:04,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17601.59 MB 2025-02-14 20:07:04,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18104.56 MB 2025-02-14 20:07:04,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 502.97 MB 2025-02-14 20:07:04,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24341.64 MB 2025-02-14 20:07:04,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20168.31 MB 2025-02-14 20:07:04,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4173.33 MB 2025-02-14 20:07:04,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22027.08 MB 2025-02-14 20:07:04,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:07:04,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:07:04,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:07:04,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:04,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18104.56 MB 2025-02-14 20:07:04,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19895.00 MB 2025-02-14 20:07:04,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1790.44 MB 2025-02-14 20:07:04,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20168.31 MB 2025-02-14 20:07:04,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22854.76 MB 2025-02-14 20:07:04,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2686.45 MB 2025-02-14 20:07:04,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21238.02 MB 2025-02-14 20:07:04,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:07:04,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:07:04,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:07:04,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:04,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19895.00 MB 2025-02-14 20:07:04,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22019.17 MB 2025-02-14 20:07:04,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2124.16 MB 2025-02-14 20:07:04,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22854.76 MB 2025-02-14 20:07:04,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29125.25 MB 2025-02-14 20:07:04,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6270.48 MB 2025-02-14 20:07:04,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27272.37 MB 2025-02-14 20:07:04,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:07:04,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:07:04,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:07:04,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:04,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18104.56 MB 2025-02-14 20:07:04,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22019.17 MB 2025-02-14 20:07:04,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3914.61 MB 2025-02-14 20:07:04,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20168.31 MB 2025-02-14 20:07:04,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29125.25 MB 2025-02-14 20:07:04,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8956.94 MB 2025-02-14 20:07:04,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27272.37 MB 2025-02-14 20:07:04,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:07:04,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:07:04,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:07:04,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:04,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23472.20 MB 2025-02-14 20:07:04,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24198.93 MB 2025-02-14 20:07:04,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 726.73 MB 2025-02-14 20:07:04,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29125.25 MB 2025-02-14 20:07:04,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29517.41 MB 2025-02-14 20:07:04,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 392.17 MB 2025-02-14 20:07:04,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24869.56 MB 2025-02-14 20:07:04,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:07:04,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:07:04,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:07:04,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:04,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.15 MB 2025-02-14 20:07:04,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24801.20 MB 2025-02-14 20:07:04,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.06 MB 2025-02-14 20:07:04,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29517.41 MB 2025-02-14 20:07:04,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29519.51 MB 2025-02-14 20:07:04,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 20:07:04,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24970.37 MB 2025-02-14 20:07:04,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:07:04,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:07:04,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.10 seconds 2025-02-14 20:07:04,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:04,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-14 20:07:04,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25002.28 MB 2025-02-14 20:07:04,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10713.10 MB 2025-02-14 20:07:04,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37880.86 MB 2025-02-14 20:07:04,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29519.51 MB 2025-02-14 20:07:04,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8361.35 MB 2025-02-14 20:07:04,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25002.28 MB 2025-02-14 20:07:05,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:07:05,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:07:05,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 20:07:05,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:05,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25002.28 MB 2025-02-14 20:07:05,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19194.06 MB 2025-02-14 20:07:05,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5808.21 MB 2025-02-14 20:07:05,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29519.51 MB 2025-02-14 20:07:05,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29519.51 MB 2025-02-14 20:07:05,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:07:05,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27815.34 MB 2025-02-14 20:07:05,252 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:07:05,253 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 20:07:05,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:07:05,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:07:05,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:07:05,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:07:05,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19194.06 MB 2025-02-14 20:07:05,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27633.09 MB 2025-02-14 20:07:05,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:07:05,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29519.51 MB 2025-02-14 20:07:05,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40009.47 MB 2025-02-14 20:07:05,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 20:07:05,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27633.09 MB 2025-02-14 20:07:05,502 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:07:05,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:07:05,505 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:07:05,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:07:05,507 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:07:05,514 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:07:05,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:07:05,517 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:07:05,517 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 20:08:18,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:08:18,157 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:08:18,165 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:08:18,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:08:18,171 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 658, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:08:18,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:08:18,173 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 658, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:08:28,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:08:28,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:08:28,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.23 seconds 2025-02-14 20:08:28,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:28,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17553.76 MB 2025-02-14 20:08:28,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19882.38 MB 2025-02-14 20:08:28,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2328.63 MB 2025-02-14 20:08:28,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52594.48 MB 2025-02-14 20:08:28,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23886.56 MB 2025-02-14 20:08:28,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28707.91 MB 2025-02-14 20:08:28,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28837.87 MB 2025-02-14 20:08:28,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:08:28,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:08:28,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 20:08:28,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:28,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19882.38 MB 2025-02-14 20:08:28,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19198.59 MB 2025-02-14 20:08:28,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -683.79 MB 2025-02-14 20:08:28,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23886.56 MB 2025-02-14 20:08:28,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31721.52 MB 2025-02-14 20:08:28,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7834.96 MB 2025-02-14 20:08:28,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28511.06 MB 2025-02-14 20:08:30,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:08:30,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:08:30,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 20:08:30,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:30,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19198.59 MB 2025-02-14 20:08:30,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19729.43 MB 2025-02-14 20:08:30,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:08:30,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31721.52 MB 2025-02-14 20:08:30,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22972.20 MB 2025-02-14 20:08:30,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8749.32 MB 2025-02-14 20:08:30,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23709.02 MB 2025-02-14 20:08:30,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:08:30,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:08:30,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:08:30,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:30,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19729.43 MB 2025-02-14 20:08:30,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21618.96 MB 2025-02-14 20:08:30,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:08:30,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22972.20 MB 2025-02-14 20:08:30,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24859.64 MB 2025-02-14 20:08:30,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:08:30,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23036.39 MB 2025-02-14 20:08:30,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:08:30,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:08:30,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:08:30,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:30,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21618.96 MB 2025-02-14 20:08:30,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23860.82 MB 2025-02-14 20:08:30,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:08:30,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24859.64 MB 2025-02-14 20:08:30,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31465.67 MB 2025-02-14 20:08:30,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 20:08:30,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.10 MB 2025-02-14 20:08:30,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:08:30,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:08:30,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:08:30,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:30,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19729.43 MB 2025-02-14 20:08:30,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23860.82 MB 2025-02-14 20:08:30,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:08:30,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22972.20 MB 2025-02-14 20:08:30,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31465.67 MB 2025-02-14 20:08:30,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 20:08:30,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.10 MB 2025-02-14 20:08:30,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:08:30,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:08:30,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 20:08:30,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:30,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25394.36 MB 2025-02-14 20:08:30,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26161.36 MB 2025-02-14 20:08:30,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:08:30,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31465.67 MB 2025-02-14 20:08:30,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31880.90 MB 2025-02-14 20:08:30,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:08:30,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26869.15 MB 2025-02-14 20:08:30,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:08:30,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:08:30,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:08:30,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:30,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26574.25 MB 2025-02-14 20:08:30,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26802.40 MB 2025-02-14 20:08:30,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.15 MB 2025-02-14 20:08:30,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31880.90 MB 2025-02-14 20:08:30,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31880.90 MB 2025-02-14 20:08:30,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:08:30,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26991.30 MB 2025-02-14 20:08:30,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:08:30,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:08:30,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.62 seconds 2025-02-14 20:08:30,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:30,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15261.23 MB 2025-02-14 20:08:30,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27002.89 MB 2025-02-14 20:08:30,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11741.65 MB 2025-02-14 20:08:30,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52594.48 MB 2025-02-14 20:08:30,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31880.90 MB 2025-02-14 20:08:30,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20713.57 MB 2025-02-14 20:08:30,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27002.89 MB 2025-02-14 20:08:31,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:08:31,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:08:31,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 20:08:31,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:31,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27002.89 MB 2025-02-14 20:08:31,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20250.42 MB 2025-02-14 20:08:31,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6752.46 MB 2025-02-14 20:08:31,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31880.90 MB 2025-02-14 20:08:31,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31880.90 MB 2025-02-14 20:08:31,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:08:31,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29501.96 MB 2025-02-14 20:08:31,083 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-14 20:08:31,084 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:08:31,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:08:31,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:08:31,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:08:31,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:08:31,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20250.42 MB 2025-02-14 20:08:31,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28647.19 MB 2025-02-14 20:08:31,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.77 MB 2025-02-14 20:08:31,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31880.90 MB 2025-02-14 20:08:31,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40229.67 MB 2025-02-14 20:08:31,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8348.76 MB 2025-02-14 20:08:31,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28647.19 MB 2025-02-14 20:08:31,255 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-14 20:08:31,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:08:31,256 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:08:31,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:08:31,257 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:08:31,262 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:08:31,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:08:31,263 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:08:31,263 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:09:24,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:09:24,976 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:09:24,981 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:09:24,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:09:24,985 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1589, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:09:24,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:09:24,986 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1589, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:09:49,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:09:49,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:09:49,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.37 seconds 2025-02-14 20:09:49,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:49,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24041.11 MB 2025-02-14 20:09:49,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29664.49 MB 2025-02-14 20:09:49,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5623.38 MB 2025-02-14 20:09:49,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52751.76 MB 2025-02-14 20:09:49,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36261.86 MB 2025-02-14 20:09:49,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16489.91 MB 2025-02-14 20:09:49,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38496.12 MB 2025-02-14 20:09:49,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:09:49,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:09:49,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 20:09:49,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:49,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29664.49 MB 2025-02-14 20:09:49,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24038.57 MB 2025-02-14 20:09:49,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5625.93 MB 2025-02-14 20:09:49,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36261.86 MB 2025-02-14 20:09:49,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52772.73 MB 2025-02-14 20:09:49,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16510.88 MB 2025-02-14 20:09:49,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45790.20 MB 2025-02-14 20:09:51,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:09:51,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:09:51,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:09:51,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:51,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24038.57 MB 2025-02-14 20:09:51,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24569.41 MB 2025-02-14 20:09:51,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:09:51,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52772.73 MB 2025-02-14 20:09:51,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32052.87 MB 2025-02-14 20:09:51,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20719.86 MB 2025-02-14 20:09:51,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28547.95 MB 2025-02-14 20:09:51,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:09:51,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:09:51,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:09:51,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:51,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24569.41 MB 2025-02-14 20:09:51,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26458.94 MB 2025-02-14 20:09:51,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:09:51,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32052.87 MB 2025-02-14 20:09:51,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32052.87 MB 2025-02-14 20:09:51,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:09:51,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27876.37 MB 2025-02-14 20:09:51,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:09:51,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:09:51,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:09:51,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:51,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26458.94 MB 2025-02-14 20:09:51,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28700.80 MB 2025-02-14 20:09:51,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:09:51,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32052.87 MB 2025-02-14 20:09:51,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36771.46 MB 2025-02-14 20:09:51,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:09:51,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34245.08 MB 2025-02-14 20:09:51,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:09:51,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:09:51,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 20:09:51,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:51,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24569.41 MB 2025-02-14 20:09:51,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28700.80 MB 2025-02-14 20:09:51,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:09:51,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32052.87 MB 2025-02-14 20:09:51,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36771.46 MB 2025-02-14 20:09:51,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:09:51,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34245.08 MB 2025-02-14 20:09:51,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:09:51,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:09:51,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:09:51,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:51,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30234.34 MB 2025-02-14 20:09:51,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31001.34 MB 2025-02-14 20:09:51,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:09:51,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36771.46 MB 2025-02-14 20:09:51,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37184.60 MB 2025-02-14 20:09:51,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 20:09:51,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31709.13 MB 2025-02-14 20:09:51,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:09:51,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:09:51,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:09:51,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:51,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31414.23 MB 2025-02-14 20:09:51,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31642.75 MB 2025-02-14 20:09:51,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-14 20:09:51,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37184.60 MB 2025-02-14 20:09:51,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37184.60 MB 2025-02-14 20:09:51,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:09:51,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31844.77 MB 2025-02-14 20:09:51,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:09:51,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:09:51,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.82 seconds 2025-02-14 20:09:51,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:51,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18504.91 MB 2025-02-14 20:09:51,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31843.23 MB 2025-02-14 20:09:51,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13338.32 MB 2025-02-14 20:09:51,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52751.76 MB 2025-02-14 20:09:51,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37184.60 MB 2025-02-14 20:09:51,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15567.16 MB 2025-02-14 20:09:51,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31844.77 MB 2025-02-14 20:09:52,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:09:52,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:09:52,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:09:52,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:52,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31843.23 MB 2025-02-14 20:09:52,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23499.44 MB 2025-02-14 20:09:52,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8343.79 MB 2025-02-14 20:09:52,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37184.60 MB 2025-02-14 20:09:52,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37184.60 MB 2025-02-14 20:09:52,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:09:52,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34346.91 MB 2025-02-14 20:09:52,091 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 20:09:52,091 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:09:52,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:09:52,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:09:52,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:09:52,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:09:52,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23499.44 MB 2025-02-14 20:09:52,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31911.87 MB 2025-02-14 20:09:52,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 20:09:52,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37184.60 MB 2025-02-14 20:09:52,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45548.04 MB 2025-02-14 20:09:52,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 20:09:52,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31911.87 MB 2025-02-14 20:09:52,256 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 20:09:52,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:09:52,257 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:09:52,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:09:52,258 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:09:52,263 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:09:52,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:09:52,264 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:09:52,264 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:10:39,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:10:39,003 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:10:39,008 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:10:39,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:10:39,012 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1241, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:10:39,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:10:39,013 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1241, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:10:58,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:10:58,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:10:58,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.04 seconds 2025-02-14 20:10:58,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:10:58,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21616.19 MB 2025-02-14 20:10:58,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.02 MB 2025-02-14 20:10:58,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4391.83 MB 2025-02-14 20:10:58,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53911.49 MB 2025-02-14 20:10:58,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35030.83 MB 2025-02-14 20:10:58,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18880.66 MB 2025-02-14 20:10:58,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34937.94 MB 2025-02-14 20:10:58,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:10:58,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:10:58,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:10:58,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:10:58,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26008.02 MB 2025-02-14 20:10:58,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22229.42 MB 2025-02-14 20:10:58,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3778.60 MB 2025-02-14 20:10:58,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35030.83 MB 2025-02-14 20:10:58,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43601.89 MB 2025-02-14 20:10:58,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8571.06 MB 2025-02-14 20:10:58,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38927.70 MB 2025-02-14 20:11:00,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:11:00,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:11:00,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:11:00,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22229.42 MB 2025-02-14 20:11:00,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22760.26 MB 2025-02-14 20:11:00,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:11:00,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43601.89 MB 2025-02-14 20:11:00,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26463.96 MB 2025-02-14 20:11:00,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17137.93 MB 2025-02-14 20:11:00,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26739.85 MB 2025-02-14 20:11:00,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:11:00,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:11:00,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:11:00,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22760.26 MB 2025-02-14 20:11:00,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24649.80 MB 2025-02-14 20:11:00,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:11:00,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26463.96 MB 2025-02-14 20:11:00,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27407.68 MB 2025-02-14 20:11:00,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 20:11:00,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26067.23 MB 2025-02-14 20:11:00,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:11:00,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:11:00,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:11:00,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24649.80 MB 2025-02-14 20:11:00,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.65 MB 2025-02-14 20:11:00,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:11:00,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27407.68 MB 2025-02-14 20:11:00,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-14 20:11:00,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 20:11:00,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32435.94 MB 2025-02-14 20:11:00,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:11:00,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:11:00,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:11:00,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22760.26 MB 2025-02-14 20:11:00,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.65 MB 2025-02-14 20:11:00,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:11:00,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26463.96 MB 2025-02-14 20:11:00,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34013.71 MB 2025-02-14 20:11:00,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 20:11:00,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32435.94 MB 2025-02-14 20:11:00,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:11:00,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:11:00,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:11:00,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28425.20 MB 2025-02-14 20:11:00,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29192.20 MB 2025-02-14 20:11:00,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:11:00,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34013.71 MB 2025-02-14 20:11:00,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34426.85 MB 2025-02-14 20:11:00,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 20:11:00,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29899.99 MB 2025-02-14 20:11:00,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:11:00,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:11:00,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:11:00,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29605.09 MB 2025-02-14 20:11:00,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29832.88 MB 2025-02-14 20:11:00,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.79 MB 2025-02-14 20:11:00,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34426.85 MB 2025-02-14 20:11:00,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34426.85 MB 2025-02-14 20:11:00,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:11:00,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30068.21 MB 2025-02-14 20:11:00,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:11:00,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:11:00,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.45 seconds 2025-02-14 20:11:00,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17292.45 MB 2025-02-14 20:11:00,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30033.36 MB 2025-02-14 20:11:00,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12740.91 MB 2025-02-14 20:11:00,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53911.49 MB 2025-02-14 20:11:00,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34426.85 MB 2025-02-14 20:11:00,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19484.64 MB 2025-02-14 20:11:00,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30068.21 MB 2025-02-14 20:11:00,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:11:00,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:11:00,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:11:00,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30033.36 MB 2025-02-14 20:11:00,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22287.34 MB 2025-02-14 20:11:00,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7746.02 MB 2025-02-14 20:11:00,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34426.85 MB 2025-02-14 20:11:00,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34426.85 MB 2025-02-14 20:11:00,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:11:00,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32537.35 MB 2025-02-14 20:11:00,752 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 20:11:00,753 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:11:00,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:11:00,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:11:00,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:11:00,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:11:00,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22287.34 MB 2025-02-14 20:11:00,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30700.86 MB 2025-02-14 20:11:00,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 20:11:00,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34426.85 MB 2025-02-14 20:11:00,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42790.29 MB 2025-02-14 20:11:00,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 20:11:00,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30700.86 MB 2025-02-14 20:11:00,914 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 20:11:00,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:11:00,916 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:11:00,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:11:00,917 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:11:00,921 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:11:00,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:11:00,922 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:11:00,923 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:12:13,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:13,871 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:12:13,876 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:12:13,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:13,880 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1002, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:12:13,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:13,882 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1002, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:12:29,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:12:29,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:12:29,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.39 seconds 2025-02-14 20:12:29,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:29,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-14 20:12:29,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23497.09 MB 2025-02-14 20:12:29,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3546.28 MB 2025-02-14 20:12:29,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51153.73 MB 2025-02-14 20:12:29,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25832.72 MB 2025-02-14 20:12:29,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25321.01 MB 2025-02-14 20:12:29,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32367.38 MB 2025-02-14 20:12:29,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:12:29,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:12:29,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:12:29,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:29,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23497.09 MB 2025-02-14 20:12:29,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20987.98 MB 2025-02-14 20:12:29,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2509.10 MB 2025-02-14 20:12:29,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25832.72 MB 2025-02-14 20:12:29,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42828.04 MB 2025-02-14 20:12:29,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16995.32 MB 2025-02-14 20:12:29,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34767.39 MB 2025-02-14 20:12:31,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:12:31,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:12:31,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:12:31,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20987.98 MB 2025-02-14 20:12:31,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21518.83 MB 2025-02-14 20:12:31,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:12:31,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42828.04 MB 2025-02-14 20:12:31,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 20:12:31,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17708.35 MB 2025-02-14 20:12:31,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25498.41 MB 2025-02-14 20:12:31,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:12:31,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:12:31,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:12:31,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21518.83 MB 2025-02-14 20:12:31,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23408.36 MB 2025-02-14 20:12:31,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:12:31,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 20:12:31,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27007.12 MB 2025-02-14 20:12:31,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:12:31,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24825.79 MB 2025-02-14 20:12:31,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:12:31,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:12:31,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:12:31,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23408.36 MB 2025-02-14 20:12:31,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25650.22 MB 2025-02-14 20:12:31,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:12:31,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27007.12 MB 2025-02-14 20:12:31,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33141.29 MB 2025-02-14 20:12:31,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:12:31,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31194.50 MB 2025-02-14 20:12:31,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:12:31,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:12:31,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:12:31,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21518.83 MB 2025-02-14 20:12:31,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25650.22 MB 2025-02-14 20:12:31,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:12:31,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 20:12:31,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33141.29 MB 2025-02-14 20:12:31,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 20:12:31,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31194.50 MB 2025-02-14 20:12:31,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:12:31,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:12:31,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:12:31,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27183.76 MB 2025-02-14 20:12:31,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27950.76 MB 2025-02-14 20:12:31,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:12:31,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33141.29 MB 2025-02-14 20:12:31,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 20:12:31,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:12:31,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28658.55 MB 2025-02-14 20:12:31,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:12:31,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:12:31,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:12:31,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28363.65 MB 2025-02-14 20:12:31,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28590.23 MB 2025-02-14 20:12:31,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.58 MB 2025-02-14 20:12:31,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33558.63 MB 2025-02-14 20:12:31,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 20:12:31,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:12:31,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28808.62 MB 2025-02-14 20:12:31,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:12:31,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:12:31,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.81 seconds 2025-02-14 20:12:31,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16459.75 MB 2025-02-14 20:12:31,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28790.71 MB 2025-02-14 20:12:31,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12330.95 MB 2025-02-14 20:12:31,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51153.73 MB 2025-02-14 20:12:31,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 20:12:31,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17595.11 MB 2025-02-14 20:12:31,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28808.62 MB 2025-02-14 20:12:31,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:12:31,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:12:31,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:12:31,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28790.71 MB 2025-02-14 20:12:31,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21446.45 MB 2025-02-14 20:12:31,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7344.26 MB 2025-02-14 20:12:31,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33558.63 MB 2025-02-14 20:12:31,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33558.63 MB 2025-02-14 20:12:31,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:12:31,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31287.63 MB 2025-02-14 20:12:31,977 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 20:12:31,977 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 20:12:31,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:12:31,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:12:31,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:12:31,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:12:31,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21446.45 MB 2025-02-14 20:12:31,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29835.59 MB 2025-02-14 20:12:31,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 20:12:31,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33558.63 MB 2025-02-14 20:12:31,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43987.76 MB 2025-02-14 20:12:31,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10429.14 MB 2025-02-14 20:12:31,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29835.59 MB 2025-02-14 20:12:32,145 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 20:12:32,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:32,146 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:12:32,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:32,147 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:12:32,152 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:12:32,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:32,153 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:12:32,153 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 20:12:42,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:42,199 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:12:42,204 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:12:42,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:42,207 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1614, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:12:42,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:12:42,208 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1614, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:13:07,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:13:07,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:13:07,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.06 seconds 2025-02-14 20:13:07,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:07,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24215.32 MB 2025-02-14 20:13:07,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29927.96 MB 2025-02-14 20:13:07,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5712.64 MB 2025-02-14 20:13:07,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52330.23 MB 2025-02-14 20:13:07,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36339.45 MB 2025-02-14 20:13:07,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15990.78 MB 2025-02-14 20:13:07,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38896.82 MB 2025-02-14 20:13:07,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:13:07,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:13:07,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 20:13:07,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:07,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29927.96 MB 2025-02-14 20:13:07,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24168.53 MB 2025-02-14 20:13:07,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5759.43 MB 2025-02-14 20:13:07,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36339.45 MB 2025-02-14 20:13:07,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54804.87 MB 2025-02-14 20:13:07,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18465.42 MB 2025-02-14 20:13:07,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46319.42 MB 2025-02-14 20:13:09,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:13:09,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:13:09,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 20:13:09,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24168.53 MB 2025-02-14 20:13:09,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24699.37 MB 2025-02-14 20:13:09,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:13:09,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54804.87 MB 2025-02-14 20:13:09,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32042.39 MB 2025-02-14 20:13:09,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22762.49 MB 2025-02-14 20:13:09,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28677.92 MB 2025-02-14 20:13:09,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:13:09,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:13:09,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:13:09,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.37 MB 2025-02-14 20:13:09,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26588.91 MB 2025-02-14 20:13:09,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:13:09,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32042.39 MB 2025-02-14 20:13:09,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32042.39 MB 2025-02-14 20:13:09,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:13:09,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28006.34 MB 2025-02-14 20:13:09,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:13:09,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:13:09,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:13:09,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26588.91 MB 2025-02-14 20:13:09,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28830.76 MB 2025-02-14 20:13:09,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:13:09,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32042.39 MB 2025-02-14 20:13:09,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36760.98 MB 2025-02-14 20:13:09,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:13:09,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34375.05 MB 2025-02-14 20:13:09,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:13:09,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:13:09,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:13:09,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.37 MB 2025-02-14 20:13:09,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28830.76 MB 2025-02-14 20:13:09,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:13:09,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32042.39 MB 2025-02-14 20:13:09,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36760.98 MB 2025-02-14 20:13:09,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:13:09,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34375.05 MB 2025-02-14 20:13:09,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:13:09,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:13:09,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:13:09,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30364.31 MB 2025-02-14 20:13:09,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31131.31 MB 2025-02-14 20:13:09,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:13:09,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36760.98 MB 2025-02-14 20:13:09,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 20:13:09,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:13:09,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.10 MB 2025-02-14 20:13:09,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:13:09,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:13:09,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:13:09,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31544.20 MB 2025-02-14 20:13:09,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31773.23 MB 2025-02-14 20:13:09,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.04 MB 2025-02-14 20:13:09,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 20:13:09,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 20:13:09,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:13:09,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31990.44 MB 2025-02-14 20:13:09,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:13:09,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:13:09,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.51 seconds 2025-02-14 20:13:09,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18592.01 MB 2025-02-14 20:13:09,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31974.31 MB 2025-02-14 20:13:09,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13382.29 MB 2025-02-14 20:13:09,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52330.23 MB 2025-02-14 20:13:09,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 20:13:09,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15151.92 MB 2025-02-14 20:13:09,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31990.44 MB 2025-02-14 20:13:09,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:13:09,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:13:09,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:13:09,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:09,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31974.31 MB 2025-02-14 20:13:09,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23596.40 MB 2025-02-14 20:13:09,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8377.90 MB 2025-02-14 20:13:09,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 20:13:09,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 20:13:09,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:13:09,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34485.97 MB 2025-02-14 20:13:10,007 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:13:10,007 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:13:10,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:13:10,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:13:10,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:13:10,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:10,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23596.40 MB 2025-02-14 20:13:10,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32035.42 MB 2025-02-14 20:13:10,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:13:10,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 20:13:10,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45569.02 MB 2025-02-14 20:13:10,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:13:10,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32035.42 MB 2025-02-14 20:13:10,169 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:13:10,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:10,171 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:13:10,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:10,172 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:13:10,176 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:13:10,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:10,177 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:13:10,177 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:13:52,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:52,693 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:13:52,697 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:13:52,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:52,701 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:13:52,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:52,702 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:13:55,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:13:55,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:13:55,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-14 20:13:55,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:55,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 20:13:55,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 20:13:55,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 20:13:55,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58154.02 MB 2025-02-14 20:13:55,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-14 20:13:55,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38149.29 MB 2025-02-14 20:13:55,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-14 20:13:55,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:13:55,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:13:55,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:13:55,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:55,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 20:13:55,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-14 20:13:55,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-14 20:13:55,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-14 20:13:55,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-14 20:13:55,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:13:55,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.37 MB 2025-02-14 20:13:55,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:13:55,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:13:55,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 20:13:55,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:55,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-14 20:13:55,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-14 20:13:55,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 20:13:55,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-14 20:13:55,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 20:13:55,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 20:13:55,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19107.09 MB 2025-02-14 20:13:56,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:13:56,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:13:56,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:13:56,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 20:13:56,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-14 20:13:56,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 20:13:56,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:13:56,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 20:13:56,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:13:56,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-14 20:13:56,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:13:56,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:13:56,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:13:56,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-14 20:13:56,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 20:13:56,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 20:13:56,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:13:56,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20677.92 MB 2025-02-14 20:13:56,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 20:13:56,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 20:13:56,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:13:56,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:13:56,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:13:56,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 20:13:56,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 20:13:56,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 20:13:56,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:13:56,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20677.92 MB 2025-02-14 20:13:56,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 20:13:56,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 20:13:56,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:13:56,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:13:56,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:13:56,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-14 20:13:56,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.71 MB 2025-02-14 20:13:56,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 20:13:56,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20677.92 MB 2025-02-14 20:13:56,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 20:13:56,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 20:13:56,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18030.52 MB 2025-02-14 20:13:56,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:13:56,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:13:56,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:13:56,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.90 MB 2025-02-14 20:13:56,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18134.12 MB 2025-02-14 20:13:56,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.21 MB 2025-02-14 20:13:56,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 20:13:56,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 20:13:56,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:13:56,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18158.46 MB 2025-02-14 20:13:56,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:13:56,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:13:56,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.47 seconds 2025-02-14 20:13:56,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 20:13:56,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18335.19 MB 2025-02-14 20:13:56,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4805.55 MB 2025-02-14 20:13:56,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58154.02 MB 2025-02-14 20:13:56,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 20:13:56,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37312.53 MB 2025-02-14 20:13:56,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18335.19 MB 2025-02-14 20:13:56,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:13:56,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:13:56,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:13:56,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18335.19 MB 2025-02-14 20:13:56,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17406.12 MB 2025-02-14 20:13:56,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -929.07 MB 2025-02-14 20:13:56,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 20:13:56,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 20:13:56,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:13:56,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19138.92 MB 2025-02-14 20:13:56,462 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:13:56,463 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:13:56,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:13:56,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:13:56,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:13:56,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:13:56,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.12 MB 2025-02-14 20:13:56,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25845.14 MB 2025-02-14 20:13:56,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:13:56,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 20:13:56,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29232.20 MB 2025-02-14 20:13:56,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:13:56,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25845.14 MB 2025-02-14 20:13:56,630 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:13:56,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:56,631 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:13:56,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:56,632 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:13:56,637 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:13:56,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:13:56,638 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:13:56,638 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:14:42,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:14:42,279 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:14:42,284 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:14:42,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:14:42,288 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:14:42,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:14:42,289 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:14:54,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:14:54,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:14:54,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.39 seconds 2025-02-14 20:14:54,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:54,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.98 MB 2025-02-14 20:14:54,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21459.49 MB 2025-02-14 20:14:54,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2860.52 MB 2025-02-14 20:14:54,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41817.21 MB 2025-02-14 20:14:54,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25170.02 MB 2025-02-14 20:14:54,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16647.19 MB 2025-02-14 20:14:54,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.08 MB 2025-02-14 20:14:54,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:14:54,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:14:54,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:14:54,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:54,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21459.49 MB 2025-02-14 20:14:54,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19978.39 MB 2025-02-14 20:14:54,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1481.10 MB 2025-02-14 20:14:54,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25170.02 MB 2025-02-14 20:14:54,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35313.94 MB 2025-02-14 20:14:54,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10143.92 MB 2025-02-14 20:14:54,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31121.93 MB 2025-02-14 20:14:56,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:14:56,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:14:56,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:14:56,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:56,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19978.39 MB 2025-02-14 20:14:56,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20509.23 MB 2025-02-14 20:14:56,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:14:56,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35313.94 MB 2025-02-14 20:14:56,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23725.08 MB 2025-02-14 20:14:56,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11588.86 MB 2025-02-14 20:14:56,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24488.82 MB 2025-02-14 20:14:56,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:14:56,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:14:56,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:14:56,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:56,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-14 20:14:56,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22398.77 MB 2025-02-14 20:14:56,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:14:56,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23725.08 MB 2025-02-14 20:14:56,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25612.52 MB 2025-02-14 20:14:56,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:14:56,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23816.20 MB 2025-02-14 20:14:56,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:14:56,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:14:56,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:14:56,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:56,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22398.77 MB 2025-02-14 20:14:56,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-14 20:14:56,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:14:56,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25612.52 MB 2025-02-14 20:14:56,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32218.55 MB 2025-02-14 20:14:56,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 20:14:56,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-14 20:14:56,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:14:56,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:14:56,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 20:14:56,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:56,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-14 20:14:56,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-14 20:14:56,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:14:56,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23725.08 MB 2025-02-14 20:14:56,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32218.55 MB 2025-02-14 20:14:56,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 20:14:56,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-14 20:14:57,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:14:57,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:14:57,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:14:57,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:57,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26174.16 MB 2025-02-14 20:14:57,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26941.17 MB 2025-02-14 20:14:57,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:14:57,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32218.55 MB 2025-02-14 20:14:57,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32633.78 MB 2025-02-14 20:14:57,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:14:57,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27648.95 MB 2025-02-14 20:14:57,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:14:57,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:14:57,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:14:57,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:57,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.06 MB 2025-02-14 20:14:57,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27583.14 MB 2025-02-14 20:14:57,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-14 20:14:57,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32633.78 MB 2025-02-14 20:14:57,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32633.78 MB 2025-02-14 20:14:57,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:14:57,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27793.76 MB 2025-02-14 20:14:57,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:14:57,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:14:57,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.84 seconds 2025-02-14 20:14:57,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:57,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15783.84 MB 2025-02-14 20:14:57,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27783.74 MB 2025-02-14 20:14:57,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11999.90 MB 2025-02-14 20:14:57,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41817.21 MB 2025-02-14 20:14:57,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32633.78 MB 2025-02-14 20:14:57,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9183.43 MB 2025-02-14 20:14:57,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27793.76 MB 2025-02-14 20:14:57,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:14:57,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:14:57,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:14:57,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:57,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27783.74 MB 2025-02-14 20:14:57,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20780.99 MB 2025-02-14 20:14:57,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7002.75 MB 2025-02-14 20:14:57,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32633.78 MB 2025-02-14 20:14:57,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32633.78 MB 2025-02-14 20:14:57,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:14:57,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30289.57 MB 2025-02-14 20:14:57,416 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 20:14:57,416 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:14:57,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:14:57,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:14:57,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:14:57,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:14:57,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20780.99 MB 2025-02-14 20:14:57,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29200.07 MB 2025-02-14 20:14:57,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 20:14:57,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32633.78 MB 2025-02-14 20:14:57,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41005.61 MB 2025-02-14 20:14:57,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 20:14:57,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29200.07 MB 2025-02-14 20:14:57,581 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 20:14:57,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:14:57,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:14:57,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:14:57,584 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:14:57,588 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:14:57,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:14:57,589 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:14:57,589 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:15:14,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:15:14,453 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:15:14,457 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:15:14,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:15:14,461 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1087, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:15:14,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:15:14,462 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1087, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:15:31,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:15:31,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:15:31,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.78 seconds 2025-02-14 20:15:31,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:31,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20543.10 MB 2025-02-14 20:15:31,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24389.93 MB 2025-02-14 20:15:31,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3846.83 MB 2025-02-14 20:15:31,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49377.44 MB 2025-02-14 20:15:31,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30339.50 MB 2025-02-14 20:15:31,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19037.95 MB 2025-02-14 20:15:31,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33268.09 MB 2025-02-14 20:15:31,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:15:31,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:15:31,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:15:31,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:31,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24389.93 MB 2025-02-14 20:15:31,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21428.83 MB 2025-02-14 20:15:31,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2961.10 MB 2025-02-14 20:15:31,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30339.50 MB 2025-02-14 20:15:31,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39168.51 MB 2025-02-14 20:15:31,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8829.01 MB 2025-02-14 20:15:31,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35234.92 MB 2025-02-14 20:15:33,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:15:33,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:15:33,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:15:33,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21428.83 MB 2025-02-14 20:15:33,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21959.67 MB 2025-02-14 20:15:33,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:15:33,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39168.51 MB 2025-02-14 20:15:33,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27906.80 MB 2025-02-14 20:15:33,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11261.71 MB 2025-02-14 20:15:33,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25938.21 MB 2025-02-14 20:15:33,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:15:33,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:15:33,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:15:33,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21959.67 MB 2025-02-14 20:15:33,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23849.20 MB 2025-02-14 20:15:33,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:15:33,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 20:15:33,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27906.80 MB 2025-02-14 20:15:33,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:15:33,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25266.63 MB 2025-02-14 20:15:33,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:15:33,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:15:33,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:15:33,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23849.20 MB 2025-02-14 20:15:33,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26091.06 MB 2025-02-14 20:15:33,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:15:33,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 20:15:33,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-14 20:15:33,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 20:15:33,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31635.34 MB 2025-02-14 20:15:33,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:15:33,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:15:33,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:15:33,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21959.67 MB 2025-02-14 20:15:33,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26091.06 MB 2025-02-14 20:15:33,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:15:33,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 20:15:33,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-14 20:15:33,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 20:15:33,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31635.34 MB 2025-02-14 20:15:33,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:15:33,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:15:33,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:15:33,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27624.60 MB 2025-02-14 20:15:33,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28391.60 MB 2025-02-14 20:15:33,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:15:33,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34043.07 MB 2025-02-14 20:15:33,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-14 20:15:33,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:15:33,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29099.39 MB 2025-02-14 20:15:33,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:15:33,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:15:33,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:15:33,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28804.49 MB 2025-02-14 20:15:33,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29033.59 MB 2025-02-14 20:15:33,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.10 MB 2025-02-14 20:15:33,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34460.40 MB 2025-02-14 20:15:33,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-14 20:15:33,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:15:33,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29272.41 MB 2025-02-14 20:15:33,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:15:33,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:15:33,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.18 seconds 2025-02-14 20:15:33,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16755.90 MB 2025-02-14 20:15:33,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29234.51 MB 2025-02-14 20:15:33,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12478.61 MB 2025-02-14 20:15:33,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49377.44 MB 2025-02-14 20:15:33,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-14 20:15:33,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14917.04 MB 2025-02-14 20:15:33,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29272.41 MB 2025-02-14 20:15:33,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:15:33,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:15:33,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:15:33,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29234.51 MB 2025-02-14 20:15:33,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21758.01 MB 2025-02-14 20:15:33,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7476.51 MB 2025-02-14 20:15:33,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34460.40 MB 2025-02-14 20:15:33,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-14 20:15:33,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:15:33,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31744.33 MB 2025-02-14 20:15:33,925 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 20:15:33,926 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:15:33,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:15:33,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:15:33,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:15:33,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:15:33,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21758.01 MB 2025-02-14 20:15:33,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30191.30 MB 2025-02-14 20:15:33,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 20:15:33,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34460.40 MB 2025-02-14 20:15:33,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42844.82 MB 2025-02-14 20:15:33,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 20:15:33,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30191.30 MB 2025-02-14 20:15:34,088 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 20:15:34,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:15:34,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:15:34,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:15:34,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:15:34,095 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:15:34,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:15:34,096 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:15:34,096 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:16:48,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:16:48,117 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:16:48,122 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:16:48,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:16:48,126 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 307, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:16:48,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:16:48,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 307, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:16:52,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:16:52,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:16:52,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.69 seconds 2025-02-14 20:16:52,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:52,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15107.93 MB 2025-02-14 20:16:52,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16194.39 MB 2025-02-14 20:16:52,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1086.46 MB 2025-02-14 20:16:52,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51229.23 MB 2025-02-14 20:16:52,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25136.46 MB 2025-02-14 20:16:52,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26092.77 MB 2025-02-14 20:16:52,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25032.29 MB 2025-02-14 20:16:52,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:16:52,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:16:52,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:16:52,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:52,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16194.39 MB 2025-02-14 20:16:52,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16713.68 MB 2025-02-14 20:16:52,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 519.30 MB 2025-02-14 20:16:52,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25136.46 MB 2025-02-14 20:16:52,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25136.46 MB 2025-02-14 20:16:52,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:16:52,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20503.10 MB 2025-02-14 20:16:54,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:16:54,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:16:54,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.45 seconds 2025-02-14 20:16:54,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16713.68 MB 2025-02-14 20:16:54,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17119.78 MB 2025-02-14 20:16:54,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.09 MB 2025-02-14 20:16:54,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25136.46 MB 2025-02-14 20:16:54,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24192.75 MB 2025-02-14 20:16:54,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 20:16:54,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21053.20 MB 2025-02-14 20:16:54,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:16:54,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:16:54,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:16:54,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17119.78 MB 2025-02-14 20:16:54,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18565.76 MB 2025-02-14 20:16:54,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.99 MB 2025-02-14 20:16:54,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24192.75 MB 2025-02-14 20:16:54,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24192.75 MB 2025-02-14 20:16:54,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:16:54,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19650.10 MB 2025-02-14 20:16:54,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:16:54,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:16:54,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:16:54,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18565.76 MB 2025-02-14 20:16:54,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20280.80 MB 2025-02-14 20:16:54,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1715.04 MB 2025-02-14 20:16:54,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24192.75 MB 2025-02-14 20:16:54,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26363.30 MB 2025-02-14 20:16:54,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2170.55 MB 2025-02-14 20:16:54,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24522.16 MB 2025-02-14 20:16:54,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:16:54,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:16:54,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 20:16:54,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17119.78 MB 2025-02-14 20:16:54,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20280.80 MB 2025-02-14 20:16:54,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3161.02 MB 2025-02-14 20:16:54,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24192.75 MB 2025-02-14 20:16:54,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26363.30 MB 2025-02-14 20:16:54,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2170.55 MB 2025-02-14 20:16:54,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24522.16 MB 2025-02-14 20:16:54,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:16:54,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:16:54,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 20:16:54,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21453.96 MB 2025-02-14 20:16:54,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22040.72 MB 2025-02-14 20:16:54,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 586.76 MB 2025-02-14 20:16:54,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26363.30 MB 2025-02-14 20:16:54,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26684.16 MB 2025-02-14 20:16:54,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 320.86 MB 2025-02-14 20:16:54,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22582.18 MB 2025-02-14 20:16:54,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:16:54,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:16:54,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:16:54,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22356.58 MB 2025-02-14 20:16:54,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22563.34 MB 2025-02-14 20:16:54,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.76 MB 2025-02-14 20:16:54,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26684.16 MB 2025-02-14 20:16:54,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26688.36 MB 2025-02-14 20:16:54,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 20:16:54,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22702.29 MB 2025-02-14 20:16:54,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:16:54,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:16:54,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.48 seconds 2025-02-14 20:16:54,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14038.32 MB 2025-02-14 20:16:54,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22764.42 MB 2025-02-14 20:16:54,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8726.10 MB 2025-02-14 20:16:54,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51229.23 MB 2025-02-14 20:16:54,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26688.36 MB 2025-02-14 20:16:54,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24540.87 MB 2025-02-14 20:16:54,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22764.42 MB 2025-02-14 20:16:54,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:16:54,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:16:54,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 20:16:54,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22764.42 MB 2025-02-14 20:16:54,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25778.45 MB 2025-02-14 20:16:54,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 20:16:54,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26688.36 MB 2025-02-14 20:16:54,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26822.57 MB 2025-02-14 20:16:54,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-14 20:16:54,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26080.08 MB 2025-02-14 20:16:54,895 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:16:54,895 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:16:54,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:16:54,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:16:54,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:16:54,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:16:54,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18599.10 MB 2025-02-14 20:16:54,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27038.12 MB 2025-02-14 20:16:54,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:16:54,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26822.57 MB 2025-02-14 20:16:54,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35213.28 MB 2025-02-14 20:16:54,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:16:54,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27038.12 MB 2025-02-14 20:16:55,063 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:16:55,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:16:55,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:16:55,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:16:55,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:16:55,070 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:16:55,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:16:55,071 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:16:55,071 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:17:58,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:17:58,979 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:17:58,987 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:17:58,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:17:58,995 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1780, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:17:58,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:17:58,997 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1780, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:18:26,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:18:26,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:18:26,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.36 seconds 2025-02-14 20:18:26,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:26,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25372.03 MB 2025-02-14 20:18:26,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31671.88 MB 2025-02-14 20:18:26,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6299.84 MB 2025-02-14 20:18:26,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47798.29 MB 2025-02-14 20:18:26,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36995.86 MB 2025-02-14 20:18:26,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10802.43 MB 2025-02-14 20:18:26,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40506.52 MB 2025-02-14 20:18:26,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:18:26,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:18:26,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 20:18:26,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:26,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31671.88 MB 2025-02-14 20:18:26,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25031.52 MB 2025-02-14 20:18:26,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6640.36 MB 2025-02-14 20:18:26,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36995.86 MB 2025-02-14 20:18:26,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59324.24 MB 2025-02-14 20:18:26,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22328.38 MB 2025-02-14 20:18:26,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50067.05 MB 2025-02-14 20:18:28,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:18:28,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:18:28,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 20:18:28,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:28,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25031.52 MB 2025-02-14 20:18:28,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25562.36 MB 2025-02-14 20:18:28,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:18:28,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59324.24 MB 2025-02-14 20:18:28,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-14 20:18:28,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31406.95 MB 2025-02-14 20:18:28,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29541.94 MB 2025-02-14 20:18:28,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:18:28,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:18:28,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:18:28,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:28,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-14 20:18:28,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27451.89 MB 2025-02-14 20:18:28,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:18:28,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 20:18:28,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30748.44 MB 2025-02-14 20:18:28,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 20:18:28,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28869.32 MB 2025-02-14 20:18:28,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:18:28,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:18:28,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:18:28,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:28,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27451.89 MB 2025-02-14 20:18:28,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-14 20:18:28,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:18:28,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30748.44 MB 2025-02-14 20:18:28,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36882.61 MB 2025-02-14 20:18:28,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:18:28,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-14 20:18:28,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:18:28,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:18:28,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:18:28,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:28,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-14 20:18:28,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-14 20:18:28,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:18:28,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 20:18:28,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36882.61 MB 2025-02-14 20:18:28,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 20:18:28,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-14 20:18:28,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:18:28,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:18:28,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:18:28,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:28,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31227.29 MB 2025-02-14 20:18:28,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31994.29 MB 2025-02-14 20:18:28,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:18:28,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36882.61 MB 2025-02-14 20:18:28,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37299.95 MB 2025-02-14 20:18:28,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:18:28,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32702.08 MB 2025-02-14 20:18:28,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:18:28,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:18:28,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:18:28,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:28,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32407.18 MB 2025-02-14 20:18:28,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32636.04 MB 2025-02-14 20:18:28,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-14 20:18:28,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37299.95 MB 2025-02-14 20:18:28,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37299.95 MB 2025-02-14 20:18:28,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:18:28,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32843.25 MB 2025-02-14 20:18:28,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:18:28,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:18:28,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.83 seconds 2025-02-14 20:18:28,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:28,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19170.37 MB 2025-02-14 20:18:28,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32836.82 MB 2025-02-14 20:18:28,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13666.45 MB 2025-02-14 20:18:28,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47798.29 MB 2025-02-14 20:18:28,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37299.95 MB 2025-02-14 20:18:28,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10498.34 MB 2025-02-14 20:18:28,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32843.25 MB 2025-02-14 20:18:29,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:18:29,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:18:29,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:18:29,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:29,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32836.82 MB 2025-02-14 20:18:29,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24170.19 MB 2025-02-14 20:18:29,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8666.63 MB 2025-02-14 20:18:29,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37299.95 MB 2025-02-14 20:18:29,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37299.95 MB 2025-02-14 20:18:29,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:18:29,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35344.80 MB 2025-02-14 20:18:29,112 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 20:18:29,112 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:18:29,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:18:29,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:18:29,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:18:29,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:18:29,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24170.19 MB 2025-02-14 20:18:29,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32596.69 MB 2025-02-14 20:18:29,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 20:18:29,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37299.95 MB 2025-02-14 20:18:29,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45678.07 MB 2025-02-14 20:18:29,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 20:18:29,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32596.69 MB 2025-02-14 20:18:29,275 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 20:18:29,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:18:29,276 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:18:29,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:18:29,277 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:18:29,282 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:18:29,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:18:29,283 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:18:29,283 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:19:37,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:19:37,018 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:19:37,024 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:19:37,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:19:37,029 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1370, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:19:37,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:19:37,029 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1370, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:19:57,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:19:57,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:19:57,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.93 seconds 2025-02-14 20:19:57,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:19:57,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22515.09 MB 2025-02-14 20:19:57,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27363.70 MB 2025-02-14 20:19:57,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4848.62 MB 2025-02-14 20:19:57,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58244.20 MB 2025-02-14 20:19:57,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35534.14 MB 2025-02-14 20:19:57,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22710.06 MB 2025-02-14 20:19:57,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.81 MB 2025-02-14 20:19:58,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:19:58,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:19:58,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:19:58,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:19:58,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27363.70 MB 2025-02-14 20:19:58,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.05 MB 2025-02-14 20:19:58,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4463.65 MB 2025-02-14 20:19:58,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35534.14 MB 2025-02-14 20:19:58,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46651.15 MB 2025-02-14 20:19:58,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11117.00 MB 2025-02-14 20:19:58,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41731.92 MB 2025-02-14 20:19:59,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:19:59,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:19:59,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 20:19:59,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:19:59,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22900.05 MB 2025-02-14 20:19:59,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23430.89 MB 2025-02-14 20:19:59,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:19:59,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46651.15 MB 2025-02-14 20:19:59,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30685.53 MB 2025-02-14 20:19:59,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15965.62 MB 2025-02-14 20:19:59,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27409.44 MB 2025-02-14 20:19:59,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:19:59,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:19:59,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:19:59,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:19:59,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 20:19:59,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25320.43 MB 2025-02-14 20:19:59,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:19:59,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30685.53 MB 2025-02-14 20:19:59,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30685.53 MB 2025-02-14 20:19:59,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:19:59,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26737.86 MB 2025-02-14 20:20:00,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:20:00,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:20:00,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:20:00,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:20:00,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25320.43 MB 2025-02-14 20:20:00,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 20:20:00,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:20:00,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30685.53 MB 2025-02-14 20:20:00,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35404.12 MB 2025-02-14 20:20:00,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:20:00,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 20:20:00,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:20:00,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:20:00,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:20:00,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:20:00,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 20:20:00,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 20:20:00,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:20:00,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30685.53 MB 2025-02-14 20:20:00,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35404.12 MB 2025-02-14 20:20:00,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:20:00,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 20:20:00,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:20:00,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:20:00,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:20:00,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:20:00,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29095.83 MB 2025-02-14 20:20:00,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29863.48 MB 2025-02-14 20:20:00,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.66 MB 2025-02-14 20:20:00,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35404.12 MB 2025-02-14 20:20:00,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35819.36 MB 2025-02-14 20:20:00,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:20:00,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30571.27 MB 2025-02-14 20:20:00,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:20:00,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:20:00,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:20:00,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:20:00,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30276.37 MB 2025-02-14 20:20:00,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30505.09 MB 2025-02-14 20:20:00,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-14 20:20:00,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35819.36 MB 2025-02-14 20:20:00,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35819.36 MB 2025-02-14 20:20:00,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:20:00,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30747.21 MB 2025-02-14 20:20:00,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:20:00,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:20:00,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.33 seconds 2025-02-14 20:20:00,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:20:00,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17741.90 MB 2025-02-14 20:20:00,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30705.72 MB 2025-02-14 20:20:00,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12963.82 MB 2025-02-14 20:20:00,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58244.20 MB 2025-02-14 20:20:00,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35819.36 MB 2025-02-14 20:20:00,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22424.85 MB 2025-02-14 20:20:00,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30747.21 MB 2025-02-14 20:20:00,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:20:00,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:20:00,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 20:20:00,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:20:00,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30705.72 MB 2025-02-14 20:20:00,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22740.08 MB 2025-02-14 20:20:00,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7965.63 MB 2025-02-14 20:20:00,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35819.36 MB 2025-02-14 20:20:00,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35819.36 MB 2025-02-14 20:20:00,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:20:00,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33211.86 MB 2025-02-14 20:20:00,650 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 20:20:00,650 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:20:00,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:20:00,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:20:00,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:20:00,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:20:00,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22740.08 MB 2025-02-14 20:20:00,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31160.86 MB 2025-02-14 20:20:00,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 20:20:00,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35819.36 MB 2025-02-14 20:20:00,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44191.19 MB 2025-02-14 20:20:00,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 20:20:00,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31160.86 MB 2025-02-14 20:20:00,811 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 20:20:00,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:20:00,813 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:20:00,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:20:00,814 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:20:00,818 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:20:00,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:20:00,820 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:20:00,820 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:20:57,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:20:57,862 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:20:57,867 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:20:57,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:20:57,871 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1593, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:20:57,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:20:57,872 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1593, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:21:22,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:21:22,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:21:22,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.54 seconds 2025-02-14 20:21:22,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:22,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24068.99 MB 2025-02-14 20:21:22,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29706.52 MB 2025-02-14 20:21:22,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5637.54 MB 2025-02-14 20:21:22,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52563.02 MB 2025-02-14 20:21:22,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36310.09 MB 2025-02-14 20:21:22,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16252.93 MB 2025-02-14 20:21:22,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38524.00 MB 2025-02-14 20:21:22,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:21:22,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:21:22,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:21:22,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:22,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29706.52 MB 2025-02-14 20:21:22,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.36 MB 2025-02-14 20:21:22,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5647.16 MB 2025-02-14 20:21:22,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36310.09 MB 2025-02-14 20:21:22,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50413.44 MB 2025-02-14 20:21:22,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14103.35 MB 2025-02-14 20:21:22,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44061.57 MB 2025-02-14 20:21:24,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:21:24,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:21:24,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 20:21:24,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:24,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.36 MB 2025-02-14 20:21:24,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24590.20 MB 2025-02-14 20:21:24,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:21:24,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50413.44 MB 2025-02-14 20:21:24,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32086.43 MB 2025-02-14 20:21:24,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18327.01 MB 2025-02-14 20:21:24,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28568.75 MB 2025-02-14 20:21:24,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:21:24,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:21:24,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:21:24,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:24,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.20 MB 2025-02-14 20:21:24,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26479.74 MB 2025-02-14 20:21:24,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:21:24,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32086.43 MB 2025-02-14 20:21:24,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32086.43 MB 2025-02-14 20:21:24,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:21:24,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27897.17 MB 2025-02-14 20:21:24,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:21:24,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:21:24,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:21:24,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:24,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26479.74 MB 2025-02-14 20:21:24,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28721.59 MB 2025-02-14 20:21:24,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:21:24,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32086.43 MB 2025-02-14 20:21:24,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36805.02 MB 2025-02-14 20:21:24,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:21:24,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.87 MB 2025-02-14 20:21:24,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:21:24,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:21:24,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 20:21:24,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:24,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.20 MB 2025-02-14 20:21:24,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28721.59 MB 2025-02-14 20:21:24,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:21:24,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32086.43 MB 2025-02-14 20:21:24,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36805.02 MB 2025-02-14 20:21:24,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:21:24,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.87 MB 2025-02-14 20:21:24,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:21:24,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:21:24,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 20:21:24,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:24,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30255.14 MB 2025-02-14 20:21:24,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31022.14 MB 2025-02-14 20:21:24,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:21:24,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36805.02 MB 2025-02-14 20:21:24,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37222.35 MB 2025-02-14 20:21:24,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:21:24,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31729.93 MB 2025-02-14 20:21:24,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:21:24,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:21:24,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:21:24,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:24,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31435.03 MB 2025-02-14 20:21:24,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31663.04 MB 2025-02-14 20:21:24,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.01 MB 2025-02-14 20:21:24,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37222.35 MB 2025-02-14 20:21:24,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37222.35 MB 2025-02-14 20:21:24,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:21:24,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31898.55 MB 2025-02-14 20:21:24,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:21:24,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:21:24,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.03 seconds 2025-02-14 20:21:24,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:24,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18518.85 MB 2025-02-14 20:21:24,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31863.52 MB 2025-02-14 20:21:24,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13344.68 MB 2025-02-14 20:21:24,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52563.02 MB 2025-02-14 20:21:24,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37222.35 MB 2025-02-14 20:21:24,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15340.67 MB 2025-02-14 20:21:24,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31898.55 MB 2025-02-14 20:21:25,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:21:25,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:21:25,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:21:25,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:25,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31863.52 MB 2025-02-14 20:21:25,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23505.54 MB 2025-02-14 20:21:25,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8357.98 MB 2025-02-14 20:21:25,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37222.35 MB 2025-02-14 20:21:25,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37222.35 MB 2025-02-14 20:21:25,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:21:25,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34360.44 MB 2025-02-14 20:21:25,189 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 20:21:25,190 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:21:25,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:21:25,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:21:25,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:21:25,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:25,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23505.54 MB 2025-02-14 20:21:25,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31894.68 MB 2025-02-14 20:21:25,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 20:21:25,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37222.35 MB 2025-02-14 20:21:25,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41393.59 MB 2025-02-14 20:21:25,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 20:21:25,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31894.68 MB 2025-02-14 20:21:25,401 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 20:21:25,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:25,404 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:21:25,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:25,406 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:21:25,412 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:21:25,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:25,414 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:21:25,414 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:21:35,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:35,472 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:21:35,476 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:21:35,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:35,480 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1307, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:21:35,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:35,481 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1307, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:21:55,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:21:55,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:21:55,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.26 seconds 2025-02-14 20:21:55,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:55,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22076.09 MB 2025-02-14 20:21:55,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26702.41 MB 2025-02-14 20:21:55,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4626.32 MB 2025-02-14 20:21:55,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49736.06 MB 2025-02-14 20:21:55,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35255.22 MB 2025-02-14 20:21:55,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14480.83 MB 2025-02-14 20:21:55,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35624.33 MB 2025-02-14 20:21:55,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:21:55,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:21:55,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:21:55,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:55,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26702.41 MB 2025-02-14 20:21:55,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22572.54 MB 2025-02-14 20:21:55,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4129.87 MB 2025-02-14 20:21:55,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35255.22 MB 2025-02-14 20:21:55,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45340.43 MB 2025-02-14 20:21:55,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10085.20 MB 2025-02-14 20:21:55,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40549.03 MB 2025-02-14 20:21:57,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:21:57,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:21:57,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 20:21:57,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:57,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22572.54 MB 2025-02-14 20:21:57,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23103.38 MB 2025-02-14 20:21:57,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:21:57,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45340.43 MB 2025-02-14 20:21:57,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26457.67 MB 2025-02-14 20:21:57,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18882.76 MB 2025-02-14 20:21:57,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27082.96 MB 2025-02-14 20:21:57,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:21:57,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:21:57,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:21:57,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:57,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23103.38 MB 2025-02-14 20:21:57,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24992.91 MB 2025-02-14 20:21:57,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:21:57,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26457.67 MB 2025-02-14 20:21:57,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28345.11 MB 2025-02-14 20:21:57,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:21:57,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26410.34 MB 2025-02-14 20:21:58,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:21:58,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:21:58,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:21:58,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:58,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24992.91 MB 2025-02-14 20:21:58,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27234.77 MB 2025-02-14 20:21:58,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:21:58,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28345.11 MB 2025-02-14 20:21:58,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34479.28 MB 2025-02-14 20:21:58,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:21:58,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32779.05 MB 2025-02-14 20:21:58,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:21:58,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:21:58,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:21:58,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:58,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23103.38 MB 2025-02-14 20:21:58,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27234.77 MB 2025-02-14 20:21:58,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:21:58,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26457.67 MB 2025-02-14 20:21:58,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34479.28 MB 2025-02-14 20:21:58,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 20:21:58,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32779.05 MB 2025-02-14 20:21:58,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:21:58,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:21:58,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:21:58,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:58,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28768.31 MB 2025-02-14 20:21:58,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29535.31 MB 2025-02-14 20:21:58,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:21:58,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34479.28 MB 2025-02-14 20:21:58,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 20:21:58,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:21:58,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30243.10 MB 2025-02-14 20:21:58,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:21:58,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:21:58,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:21:58,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:58,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29948.20 MB 2025-02-14 20:21:58,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30175.20 MB 2025-02-14 20:21:58,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.99 MB 2025-02-14 20:21:58,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34894.51 MB 2025-02-14 20:21:58,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 20:21:58,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:21:58,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30419.46 MB 2025-02-14 20:21:58,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:21:58,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:21:58,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.71 seconds 2025-02-14 20:21:58,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:58,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17522.40 MB 2025-02-14 20:21:58,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30375.68 MB 2025-02-14 20:21:58,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12853.28 MB 2025-02-14 20:21:58,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49736.06 MB 2025-02-14 20:21:58,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 20:21:58,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14841.54 MB 2025-02-14 20:21:58,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30419.46 MB 2025-02-14 20:21:58,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:21:58,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:21:58,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:21:58,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:58,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30375.68 MB 2025-02-14 20:21:58,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22513.73 MB 2025-02-14 20:21:58,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7861.95 MB 2025-02-14 20:21:58,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34894.51 MB 2025-02-14 20:21:58,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 20:21:58,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:21:58,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32876.59 MB 2025-02-14 20:21:58,478 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 20:21:58,478 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:21:58,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:21:58,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:21:58,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:21:58,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:21:58,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22513.73 MB 2025-02-14 20:21:58,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30917.29 MB 2025-02-14 20:21:58,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 20:21:58,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34894.51 MB 2025-02-14 20:21:58,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43249.57 MB 2025-02-14 20:21:58,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 20:21:58,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30917.29 MB 2025-02-14 20:21:58,639 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 20:21:58,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:58,641 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:21:58,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:58,642 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:21:58,646 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:21:58,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:21:58,647 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:21:58,647 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:22:52,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:22:52,569 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:22:52,576 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:22:52,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:22:52,583 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 196, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:22:52,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:22:52,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 196, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:22:55,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:22:55,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:22:55,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.17 seconds 2025-02-14 20:22:55,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:55,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14334.47 MB 2025-02-14 20:22:55,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15028.10 MB 2025-02-14 20:22:55,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 693.63 MB 2025-02-14 20:22:55,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51604.62 MB 2025-02-14 20:22:55,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 20:22:55,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32071.75 MB 2025-02-14 20:22:55,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24032.33 MB 2025-02-14 20:22:55,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:22:55,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:22:55,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:22:55,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:55,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15028.10 MB 2025-02-14 20:22:55,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15237.75 MB 2025-02-14 20:22:55,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.65 MB 2025-02-14 20:22:55,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:22:55,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 20:22:55,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:22:55,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17543.39 MB 2025-02-14 20:22:56,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:22:56,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:22:56,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.87 seconds 2025-02-14 20:22:56,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:56,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15237.75 MB 2025-02-14 20:22:56,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15473.97 MB 2025-02-14 20:22:56,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.22 MB 2025-02-14 20:22:56,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:22:56,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 20:22:56,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:22:56,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19408.44 MB 2025-02-14 20:22:56,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:22:56,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:22:56,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:22:56,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:56,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15473.91 MB 2025-02-14 20:22:56,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16314.55 MB 2025-02-14 20:22:56,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.64 MB 2025-02-14 20:22:56,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:22:56,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 20:22:56,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:22:56,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16945.31 MB 2025-02-14 20:22:56,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:22:56,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:22:56,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:22:56,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:56,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16314.55 MB 2025-02-14 20:22:56,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17312.21 MB 2025-02-14 20:22:56,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.66 MB 2025-02-14 20:22:56,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:22:56,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21218.98 MB 2025-02-14 20:22:56,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1686.11 MB 2025-02-14 20:22:56,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.38 MB 2025-02-14 20:22:56,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:22:56,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:22:56,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 20:22:56,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:56,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15473.91 MB 2025-02-14 20:22:56,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17312.21 MB 2025-02-14 20:22:56,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1838.30 MB 2025-02-14 20:22:56,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 20:22:56,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21218.98 MB 2025-02-14 20:22:56,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1686.11 MB 2025-02-14 20:22:56,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.38 MB 2025-02-14 20:22:56,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:22:56,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:22:56,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:22:56,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:56,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17994.63 MB 2025-02-14 20:22:56,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18336.29 MB 2025-02-14 20:22:56,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.66 MB 2025-02-14 20:22:56,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21218.98 MB 2025-02-14 20:22:56,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21399.34 MB 2025-02-14 20:22:56,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 20:22:56,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18655.48 MB 2025-02-14 20:22:56,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:22:56,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:22:56,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:22:56,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:56,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18520.03 MB 2025-02-14 20:22:56,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18724.79 MB 2025-02-14 20:22:56,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.75 MB 2025-02-14 20:22:56,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21399.34 MB 2025-02-14 20:22:56,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21403.53 MB 2025-02-14 20:22:56,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 20:22:56,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18761.91 MB 2025-02-14 20:22:56,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:22:56,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:22:56,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.35 seconds 2025-02-14 20:22:56,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:56,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13651.59 MB 2025-02-14 20:22:56,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18925.86 MB 2025-02-14 20:22:56,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5274.27 MB 2025-02-14 20:22:56,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51604.62 MB 2025-02-14 20:22:56,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21403.53 MB 2025-02-14 20:22:56,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30201.09 MB 2025-02-14 20:22:56,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18925.86 MB 2025-02-14 20:22:57,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:22:57,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:22:57,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 20:22:57,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:57,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18925.86 MB 2025-02-14 20:22:57,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17608.63 MB 2025-02-14 20:22:57,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1317.23 MB 2025-02-14 20:22:57,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21403.53 MB 2025-02-14 20:22:57,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21403.53 MB 2025-02-14 20:22:57,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:22:57,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19126.82 MB 2025-02-14 20:22:57,251 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:22:57,252 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:22:57,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:22:57,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:22:57,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:22:57,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:22:57,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17608.63 MB 2025-02-14 20:22:57,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26047.66 MB 2025-02-14 20:22:57,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:22:57,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21403.53 MB 2025-02-14 20:22:57,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29794.24 MB 2025-02-14 20:22:57,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:22:57,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26047.66 MB 2025-02-14 20:22:57,506 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:22:57,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:22:57,509 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:22:57,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:22:57,510 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:22:57,518 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:22:57,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:22:57,520 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:22:57,520 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:24:03,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:03,479 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:24:03,484 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:24:03,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:03,488 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1250, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:24:03,489 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:03,489 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1250, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:24:22,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:24:22,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:24:22,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.13 seconds 2025-02-14 20:24:22,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:22,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21678.91 MB 2025-02-14 20:24:22,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26102.59 MB 2025-02-14 20:24:22,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4423.68 MB 2025-02-14 20:24:22,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42379.25 MB 2025-02-14 20:24:22,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35123.10 MB 2025-02-14 20:24:22,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7256.15 MB 2025-02-14 20:24:22,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35000.65 MB 2025-02-14 20:24:22,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:24:22,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:24:22,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:24:22,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:22,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26102.59 MB 2025-02-14 20:24:22,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22276.21 MB 2025-02-14 20:24:22,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3826.38 MB 2025-02-14 20:24:22,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35123.10 MB 2025-02-14 20:24:22,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43847.25 MB 2025-02-14 20:24:22,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8724.15 MB 2025-02-14 20:24:22,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39228.06 MB 2025-02-14 20:24:24,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:24:24,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:24:24,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:24:24,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:24,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22276.21 MB 2025-02-14 20:24:24,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22807.05 MB 2025-02-14 20:24:24,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:24:24,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43847.25 MB 2025-02-14 20:24:24,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26503.81 MB 2025-02-14 20:24:24,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17343.45 MB 2025-02-14 20:24:24,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26786.64 MB 2025-02-14 20:24:24,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:24:24,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:24:24,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:24:24,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:24,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22807.05 MB 2025-02-14 20:24:24,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24696.59 MB 2025-02-14 20:24:24,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:24:24,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26503.81 MB 2025-02-14 20:24:24,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27447.53 MB 2025-02-14 20:24:24,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 20:24:24,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26114.02 MB 2025-02-14 20:24:24,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:24:24,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:24:24,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:24:24,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:24,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24696.59 MB 2025-02-14 20:24:24,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26938.44 MB 2025-02-14 20:24:24,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:24:24,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27447.53 MB 2025-02-14 20:24:24,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34053.55 MB 2025-02-14 20:24:24,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 20:24:24,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32482.72 MB 2025-02-14 20:24:24,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:24:24,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:24:24,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 20:24:24,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:24,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22807.05 MB 2025-02-14 20:24:24,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26938.44 MB 2025-02-14 20:24:24,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:24:24,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26503.81 MB 2025-02-14 20:24:24,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34053.55 MB 2025-02-14 20:24:24,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 20:24:24,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32482.72 MB 2025-02-14 20:24:25,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:24:25,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:24:25,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:24:25,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:25,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28471.98 MB 2025-02-14 20:24:25,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29238.99 MB 2025-02-14 20:24:25,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:24:25,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34053.55 MB 2025-02-14 20:24:25,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 20:24:25,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:24:25,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29946.78 MB 2025-02-14 20:24:25,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:24:25,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:24:25,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:24:25,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:25,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29651.88 MB 2025-02-14 20:24:25,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29880.34 MB 2025-02-14 20:24:25,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 20:24:25,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 20:24:25,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 20:24:25,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:24:25,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30112.33 MB 2025-02-14 20:24:25,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:24:25,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:24:25,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.55 seconds 2025-02-14 20:24:25,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:25,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17323.81 MB 2025-02-14 20:24:25,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30080.73 MB 2025-02-14 20:24:25,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12756.92 MB 2025-02-14 20:24:25,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42379.25 MB 2025-02-14 20:24:25,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 20:24:25,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7910.46 MB 2025-02-14 20:24:25,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30112.33 MB 2025-02-14 20:24:25,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:24:25,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:24:25,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:24:25,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:25,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30080.73 MB 2025-02-14 20:24:25,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22317.63 MB 2025-02-14 20:24:25,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7763.10 MB 2025-02-14 20:24:25,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 20:24:25,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 20:24:25,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:24:25,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32583.89 MB 2025-02-14 20:24:25,327 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 20:24:25,328 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:24:25,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:24:25,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:24:25,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:24:25,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:25,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22317.63 MB 2025-02-14 20:24:25,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30727.42 MB 2025-02-14 20:24:25,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.79 MB 2025-02-14 20:24:25,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 20:24:25,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38650.51 MB 2025-02-14 20:24:25,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 20:24:25,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30727.42 MB 2025-02-14 20:24:25,493 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 20:24:25,494 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:25,494 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:24:25,495 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:25,495 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:24:25,500 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:24:25,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:25,501 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:24:25,501 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:24:33,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:33,606 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:24:33,614 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:24:33,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:33,621 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1484, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:24:33,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:33,623 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1484, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:24:56,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:24:56,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:24:56,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.12 seconds 2025-02-14 20:24:56,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:56,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23309.46 MB 2025-02-14 20:24:56,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28561.25 MB 2025-02-14 20:24:56,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5251.79 MB 2025-02-14 20:24:56,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47009.76 MB 2025-02-14 20:24:56,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35907.44 MB 2025-02-14 20:24:56,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11102.32 MB 2025-02-14 20:24:56,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37537.97 MB 2025-02-14 20:24:56,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:24:56,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:24:56,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:24:56,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:56,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28561.25 MB 2025-02-14 20:24:56,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23492.70 MB 2025-02-14 20:24:56,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5068.55 MB 2025-02-14 20:24:56,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35907.44 MB 2025-02-14 20:24:56,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49746.54 MB 2025-02-14 20:24:56,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13839.11 MB 2025-02-14 20:24:56,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44277.72 MB 2025-02-14 20:24:58,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:24:58,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:24:58,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 20:24:58,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:58,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23492.70 MB 2025-02-14 20:24:58,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24023.55 MB 2025-02-14 20:24:58,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:24:58,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49746.54 MB 2025-02-14 20:24:58,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30654.07 MB 2025-02-14 20:24:58,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19092.47 MB 2025-02-14 20:24:58,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28002.09 MB 2025-02-14 20:24:58,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:24:58,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:24:58,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:24:58,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:58,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24023.55 MB 2025-02-14 20:24:58,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25913.08 MB 2025-02-14 20:24:58,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:24:58,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30654.07 MB 2025-02-14 20:24:58,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30654.07 MB 2025-02-14 20:24:58,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:24:58,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27330.51 MB 2025-02-14 20:24:58,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:24:58,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:24:58,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:24:58,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:58,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25913.08 MB 2025-02-14 20:24:58,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28154.94 MB 2025-02-14 20:24:58,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:24:58,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30654.07 MB 2025-02-14 20:24:58,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36316.38 MB 2025-02-14 20:24:58,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 20:24:58,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33699.22 MB 2025-02-14 20:24:58,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:24:58,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:24:58,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:24:58,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:58,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24023.55 MB 2025-02-14 20:24:58,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28154.94 MB 2025-02-14 20:24:58,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:24:58,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30654.07 MB 2025-02-14 20:24:58,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36316.38 MB 2025-02-14 20:24:58,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 20:24:58,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33699.22 MB 2025-02-14 20:24:59,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:24:59,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:24:59,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:24:59,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:59,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.48 MB 2025-02-14 20:24:59,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30455.48 MB 2025-02-14 20:24:59,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:24:59,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36316.38 MB 2025-02-14 20:24:59,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36733.71 MB 2025-02-14 20:24:59,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:24:59,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31163.27 MB 2025-02-14 20:24:59,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:24:59,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:24:59,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:24:59,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:59,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30868.37 MB 2025-02-14 20:24:59,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31096.76 MB 2025-02-14 20:24:59,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-14 20:24:59,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36733.71 MB 2025-02-14 20:24:59,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36733.71 MB 2025-02-14 20:24:59,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:24:59,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31330.78 MB 2025-02-14 20:24:59,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:24:59,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:24:59,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.56 seconds 2025-02-14 20:24:59,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:59,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18139.08 MB 2025-02-14 20:24:59,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31297.08 MB 2025-02-14 20:24:59,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13157.99 MB 2025-02-14 20:24:59,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47009.76 MB 2025-02-14 20:24:59,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36733.71 MB 2025-02-14 20:24:59,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10276.04 MB 2025-02-14 20:24:59,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31330.78 MB 2025-02-14 20:24:59,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:24:59,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:24:59,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:24:59,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:59,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31297.08 MB 2025-02-14 20:24:59,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23131.83 MB 2025-02-14 20:24:59,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8165.24 MB 2025-02-14 20:24:59,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36733.71 MB 2025-02-14 20:24:59,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36733.71 MB 2025-02-14 20:24:59,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:24:59,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33799.22 MB 2025-02-14 20:24:59,472 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 20:24:59,472 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 20:24:59,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:24:59,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:24:59,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:24:59,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:24:59,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23131.83 MB 2025-02-14 20:24:59,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31539.57 MB 2025-02-14 20:24:59,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-14 20:24:59,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36733.71 MB 2025-02-14 20:24:59,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45092.96 MB 2025-02-14 20:24:59,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 20:24:59,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31539.57 MB 2025-02-14 20:24:59,635 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 20:24:59,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:59,637 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:24:59,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:59,638 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:24:59,642 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:24:59,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:24:59,643 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:24:59,644 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 20:26:26,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:26,405 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:26:26,410 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:26:26,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:26,414 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 86, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:26:26,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:26,415 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 86, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:26:27,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:26:27,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:26:27,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.34 seconds 2025-02-14 20:26:27,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:27,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-14 20:26:27,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13872.32 MB 2025-02-14 20:26:27,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.35 MB 2025-02-14 20:26:27,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53452.21 MB 2025-02-14 20:26:27,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:26:27,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34395.39 MB 2025-02-14 20:26:27,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22812.85 MB 2025-02-14 20:26:27,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:26:27,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:26:27,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:26:27,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:27,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13872.32 MB 2025-02-14 20:26:27,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14019.77 MB 2025-02-14 20:26:27,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.46 MB 2025-02-14 20:26:27,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:26:27,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:26:27,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:26:27,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14476.36 MB 2025-02-14 20:26:28,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:26:28,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:26:28,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.42 seconds 2025-02-14 20:26:28,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14019.77 MB 2025-02-14 20:26:28,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14133.90 MB 2025-02-14 20:26:28,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 114.13 MB 2025-02-14 20:26:28,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:26:28,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:26:28,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:26:28,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18104.49 MB 2025-02-14 20:26:28,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:26:28,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:26:28,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:26:28,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 20:26:28,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14539.99 MB 2025-02-14 20:26:28,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.15 MB 2025-02-14 20:26:28,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:26:28,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:26:28,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:26:28,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.74 MB 2025-02-14 20:26:28,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:26:28,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:26:28,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:26:28,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14539.99 MB 2025-02-14 20:26:28,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 20:26:28,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 493.32 MB 2025-02-14 20:26:28,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:26:28,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:26:28,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:26:28,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-14 20:26:28,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:26:28,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:26:28,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:26:28,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 20:26:28,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 20:26:28,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 899.47 MB 2025-02-14 20:26:28,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:26:28,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:26:28,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:26:28,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-14 20:26:28,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:26:28,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:26:28,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 20:26:28,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15509.56 MB 2025-02-14 20:26:28,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15716.74 MB 2025-02-14 20:26:28,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.18 MB 2025-02-14 20:26:28,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:26:28,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19188.94 MB 2025-02-14 20:26:28,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 20:26:28,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15868.91 MB 2025-02-14 20:26:28,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:26:28,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:26:28,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:26:28,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15847.79 MB 2025-02-14 20:26:28,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16053.64 MB 2025-02-14 20:26:28,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.85 MB 2025-02-14 20:26:28,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19188.94 MB 2025-02-14 20:26:28,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19188.94 MB 2025-02-14 20:26:28,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:26:28,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16053.64 MB 2025-02-14 20:26:28,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:26:28,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:26:28,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:26:28,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13268.34 MB 2025-02-14 20:26:28,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16237.13 MB 2025-02-14 20:26:28,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2968.79 MB 2025-02-14 20:26:28,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53452.21 MB 2025-02-14 20:26:28,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19188.94 MB 2025-02-14 20:26:28,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34263.27 MB 2025-02-14 20:26:28,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16237.13 MB 2025-02-14 20:26:28,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:26:28,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:26:28,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 20:26:28,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13768.03 MB 2025-02-14 20:26:28,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16518.48 MB 2025-02-14 20:26:28,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2750.45 MB 2025-02-14 20:26:28,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19188.94 MB 2025-02-14 20:26:28,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19188.94 MB 2025-02-14 20:26:28,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:26:28,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16793.49 MB 2025-02-14 20:26:28,597 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7447, cut from 7449 2025-02-14 20:26:28,597 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:26:28,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:26:28,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:26:28,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:26:28,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:26:28,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16518.48 MB 2025-02-14 20:26:28,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24219.30 MB 2025-02-14 20:26:28,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7700.82 MB 2025-02-14 20:26:28,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19188.94 MB 2025-02-14 20:26:28,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26845.64 MB 2025-02-14 20:26:28,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7656.70 MB 2025-02-14 20:26:28,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24219.30 MB 2025-02-14 20:26:28,751 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7239] 2025-02-14 20:26:28,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:28,752 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:26:28,753 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:28,753 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:26:28,758 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:26:28,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:28,759 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:26:28,759 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:26:38,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:38,627 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:26:38,632 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:26:38,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:38,636 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2057, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:26:38,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:26:38,637 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2057, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:27:10,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:27:10,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:27:10,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.69 seconds 2025-02-14 20:27:10,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:10,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27302.21 MB 2025-02-14 20:27:10,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34581.82 MB 2025-02-14 20:27:10,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7279.61 MB 2025-02-14 20:27:10,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38329.65 MB 2025-02-14 20:27:10,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40705.72 MB 2025-02-14 20:27:10,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2376.07 MB 2025-02-14 20:27:10,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43568.35 MB 2025-02-14 20:27:10,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:27:10,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:27:10,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 20:27:10,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:10,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34581.82 MB 2025-02-14 20:27:10,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26471.55 MB 2025-02-14 20:27:10,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8110.27 MB 2025-02-14 20:27:10,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40705.72 MB 2025-02-14 20:27:10,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65166.90 MB 2025-02-14 20:27:10,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24461.18 MB 2025-02-14 20:27:10,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56172.08 MB 2025-02-14 20:27:12,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:27:12,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:27:12,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 20:27:12,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:12,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26471.55 MB 2025-02-14 20:27:12,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27002.39 MB 2025-02-14 20:27:12,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:27:12,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65166.90 MB 2025-02-14 20:27:12,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31012.68 MB 2025-02-14 20:27:12,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34154.22 MB 2025-02-14 20:27:12,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30981.98 MB 2025-02-14 20:27:12,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:27:12,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:27:12,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:27:12,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:12,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27002.39 MB 2025-02-14 20:27:12,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28891.93 MB 2025-02-14 20:27:12,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:27:12,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31012.68 MB 2025-02-14 20:27:12,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32900.12 MB 2025-02-14 20:27:12,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:27:12,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30309.36 MB 2025-02-14 20:27:12,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:27:12,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:27:12,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:27:12,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:12,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28891.93 MB 2025-02-14 20:27:12,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31133.78 MB 2025-02-14 20:27:12,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:27:12,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32900.12 MB 2025-02-14 20:27:12,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38562.43 MB 2025-02-14 20:27:12,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 20:27:12,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36678.07 MB 2025-02-14 20:27:12,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:27:12,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:27:12,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:27:12,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:12,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27002.39 MB 2025-02-14 20:27:12,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31133.78 MB 2025-02-14 20:27:12,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:27:12,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31012.68 MB 2025-02-14 20:27:12,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38562.43 MB 2025-02-14 20:27:12,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 20:27:12,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36678.07 MB 2025-02-14 20:27:12,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:27:12,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:27:12,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:27:12,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:12,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32667.33 MB 2025-02-14 20:27:12,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33434.33 MB 2025-02-14 20:27:12,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:27:12,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38562.43 MB 2025-02-14 20:27:12,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38979.76 MB 2025-02-14 20:27:12,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:27:12,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34142.12 MB 2025-02-14 20:27:12,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:27:12,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:27:12,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:27:12,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:12,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33847.22 MB 2025-02-14 20:27:12,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34075.93 MB 2025-02-14 20:27:12,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-14 20:27:12,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38979.76 MB 2025-02-14 20:27:12,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38979.76 MB 2025-02-14 20:27:12,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:12,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34321.18 MB 2025-02-14 20:27:12,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:27:12,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:27:12,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.20 seconds 2025-02-14 20:27:12,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:12,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20135.46 MB 2025-02-14 20:27:12,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34276.56 MB 2025-02-14 20:27:12,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14141.10 MB 2025-02-14 20:27:12,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38329.65 MB 2025-02-14 20:27:12,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38979.76 MB 2025-02-14 20:27:12,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 650.12 MB 2025-02-14 20:27:12,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34321.18 MB 2025-02-14 20:27:13,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:27:13,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:27:13,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:27:13,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:13,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34276.56 MB 2025-02-14 20:27:13,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25132.99 MB 2025-02-14 20:27:13,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9143.57 MB 2025-02-14 20:27:13,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38979.76 MB 2025-02-14 20:27:13,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38979.76 MB 2025-02-14 20:27:13,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:13,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36782.70 MB 2025-02-14 20:27:13,125 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 20:27:13,125 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:27:13,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:27:13,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:27:13,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:27:13,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:13,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25132.99 MB 2025-02-14 20:27:13,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33553.77 MB 2025-02-14 20:27:13,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 20:27:13,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38979.76 MB 2025-02-14 20:27:13,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47351.60 MB 2025-02-14 20:27:13,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 20:27:13,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33553.77 MB 2025-02-14 20:27:13,293 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 20:27:13,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:13,295 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:27:13,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:13,296 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:27:13,301 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:27:13,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:13,302 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:27:13,302 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:27:22,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:22,564 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:27:22,569 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:27:22,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:22,572 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 175, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:27:22,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:22,573 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 175, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:27:25,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:27:25,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:27:25,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.73 seconds 2025-02-14 20:27:25,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:25,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14188.43 MB 2025-02-14 20:27:25,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14807.75 MB 2025-02-14 20:27:25,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 619.32 MB 2025-02-14 20:27:25,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55723.43 MB 2025-02-14 20:27:25,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:25,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32839.30 MB 2025-02-14 20:27:25,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23659.80 MB 2025-02-14 20:27:25,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:27:25,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:27:25,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:27:25,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:25,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14807.75 MB 2025-02-14 20:27:25,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15107.80 MB 2025-02-14 20:27:25,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 300.06 MB 2025-02-14 20:27:25,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:25,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:25,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:25,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17318.95 MB 2025-02-14 20:27:26,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:27:26,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:27:26,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 20:27:26,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15107.80 MB 2025-02-14 20:27:26,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15340.05 MB 2025-02-14 20:27:26,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.24 MB 2025-02-14 20:27:26,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:26,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:26,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:26,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19277.45 MB 2025-02-14 20:27:26,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:27:26,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:27:26,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:27:26,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15339.98 MB 2025-02-14 20:27:26,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16166.45 MB 2025-02-14 20:27:26,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 826.47 MB 2025-02-14 20:27:26,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:26,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:26,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:26,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16786.58 MB 2025-02-14 20:27:26,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:27:26,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:27:26,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:27:26,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16166.45 MB 2025-02-14 20:27:26,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17148.18 MB 2025-02-14 20:27:26,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 981.73 MB 2025-02-14 20:27:26,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:26,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:26,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:26,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19573.77 MB 2025-02-14 20:27:26,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:27:26,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:27:26,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 20:27:26,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15339.98 MB 2025-02-14 20:27:26,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17148.18 MB 2025-02-14 20:27:26,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1808.20 MB 2025-02-14 20:27:26,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:26,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:26,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:26,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19573.77 MB 2025-02-14 20:27:26,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:27:26,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:27:26,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:27:26,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17819.99 MB 2025-02-14 20:27:26,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18155.56 MB 2025-02-14 20:27:26,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 335.56 MB 2025-02-14 20:27:26,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:26,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 20:27:26,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 20:27:26,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18471.84 MB 2025-02-14 20:27:26,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:27:26,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:27:26,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:27:26,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18336.20 MB 2025-02-14 20:27:26,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18561.04 MB 2025-02-14 20:27:26,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.84 MB 2025-02-14 20:27:26,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 20:27:26,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 20:27:26,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:26,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18592.97 MB 2025-02-14 20:27:26,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:27:26,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:27:26,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.78 seconds 2025-02-14 20:27:26,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13578.72 MB 2025-02-14 20:27:26,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18762.02 MB 2025-02-14 20:27:26,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5183.30 MB 2025-02-14 20:27:26,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55723.43 MB 2025-02-14 20:27:26,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 20:27:26,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32658.95 MB 2025-02-14 20:27:26,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18762.02 MB 2025-02-14 20:27:26,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:27:26,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:27:26,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:27:26,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18762.02 MB 2025-02-14 20:27:26,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17520.63 MB 2025-02-14 20:27:26,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1241.39 MB 2025-02-14 20:27:26,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 20:27:26,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 20:27:26,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:26,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18996.33 MB 2025-02-14 20:27:26,637 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 20:27:26,637 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:27:26,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:27:26,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:27:26,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:27:26,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:26,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17520.63 MB 2025-02-14 20:27:26,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25955.48 MB 2025-02-14 20:27:26,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 20:27:26,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 20:27:26,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31450.99 MB 2025-02-14 20:27:26,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 20:27:26,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25955.48 MB 2025-02-14 20:27:26,803 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 20:27:26,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:26,804 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:27:26,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:26,805 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:27:26,809 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:27:26,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:26,811 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:27:26,811 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:27:32,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:32,959 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:27:32,964 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:27:32,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:32,968 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:27:32,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:32,969 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:27:35,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:27:35,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:27:35,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-14 20:27:35,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:35,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 20:27:35,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 20:27:35,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 20:27:35,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44029.71 MB 2025-02-14 20:27:35,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:35,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21145.58 MB 2025-02-14 20:27:35,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-14 20:27:35,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:27:35,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:27:35,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:27:35,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:35,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 20:27:35,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-14 20:27:35,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-14 20:27:35,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:35,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:35,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:35,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.75 MB 2025-02-14 20:27:36,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:27:36,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:27:36,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 20:27:36,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-14 20:27:36,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-14 20:27:36,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 20:27:36,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:36,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:36,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:36,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19106.05 MB 2025-02-14 20:27:36,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:27:36,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:27:36,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:27:36,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 20:27:36,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-14 20:27:36,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 20:27:36,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:36,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:36,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:36,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-14 20:27:36,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:27:36,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:27:36,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:27:36,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-14 20:27:36,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 20:27:36,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 20:27:36,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:36,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:36,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:36,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 20:27:36,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:27:36,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:27:36,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:27:36,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 20:27:36,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 20:27:36,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 20:27:36,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:36,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22884.12 MB 2025-02-14 20:27:36,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:36,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 20:27:36,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:27:36,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:27:36,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:27:36,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-14 20:27:36,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.71 MB 2025-02-14 20:27:36,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 20:27:36,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22884.12 MB 2025-02-14 20:27:36,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23049.80 MB 2025-02-14 20:27:36,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 20:27:36,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18032.04 MB 2025-02-14 20:27:36,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:27:36,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:27:36,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:27:36,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.90 MB 2025-02-14 20:27:36,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18132.65 MB 2025-02-14 20:27:36,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.75 MB 2025-02-14 20:27:36,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23049.80 MB 2025-02-14 20:27:36,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23049.80 MB 2025-02-14 20:27:36,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:36,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18146.98 MB 2025-02-14 20:27:36,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:27:36,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:27:36,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.47 seconds 2025-02-14 20:27:36,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 20:27:36,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18333.43 MB 2025-02-14 20:27:36,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4803.79 MB 2025-02-14 20:27:36,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44029.71 MB 2025-02-14 20:27:36,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23049.80 MB 2025-02-14 20:27:36,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20979.91 MB 2025-02-14 20:27:36,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18333.43 MB 2025-02-14 20:27:36,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:27:36,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:27:36,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:27:36,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18333.43 MB 2025-02-14 20:27:36,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17401.55 MB 2025-02-14 20:27:36,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -931.88 MB 2025-02-14 20:27:36,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23049.80 MB 2025-02-14 20:27:36,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23049.80 MB 2025-02-14 20:27:36,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:27:36,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19135.99 MB 2025-02-14 20:27:36,727 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 20:27:36,727 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:27:36,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:27:36,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:27:36,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:27:36,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:27:36,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17401.55 MB 2025-02-14 20:27:36,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25828.05 MB 2025-02-14 20:27:36,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 20:27:36,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23049.80 MB 2025-02-14 20:27:36,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31427.92 MB 2025-02-14 20:27:36,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 20:27:36,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25828.05 MB 2025-02-14 20:27:36,893 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 20:27:36,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:36,895 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:27:36,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:36,896 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:27:36,900 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:27:36,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:27:36,901 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:27:36,901 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:28:10,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:28:10,756 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:28:10,761 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:28:10,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:28:10,765 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 88, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:28:10,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:28:10,765 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 88, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:28:12,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:28:12,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:28:12,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.37 seconds 2025-02-14 20:28:12,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13581.90 MB 2025-02-14 20:28:12,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13893.33 MB 2025-02-14 20:28:12,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.43 MB 2025-02-14 20:28:12,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43994.05 MB 2025-02-14 20:28:12,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:28:12,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24937.23 MB 2025-02-14 20:28:12,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22826.78 MB 2025-02-14 20:28:12,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:28:12,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:28:12,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:28:12,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13893.33 MB 2025-02-14 20:28:12,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14044.22 MB 2025-02-14 20:28:12,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 150.89 MB 2025-02-14 20:28:12,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:28:12,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:28:12,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:28:12,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14511.42 MB 2025-02-14 20:28:12,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:28:12,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:28:12,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.43 seconds 2025-02-14 20:28:12,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14044.22 MB 2025-02-14 20:28:12,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14161.00 MB 2025-02-14 20:28:12,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 116.79 MB 2025-02-14 20:28:12,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:28:12,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:28:12,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:28:12,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18128.93 MB 2025-02-14 20:28:12,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:28:12,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:28:12,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:28:12,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14160.94 MB 2025-02-14 20:28:12,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.53 MB 2025-02-14 20:28:12,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 415.60 MB 2025-02-14 20:28:12,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:28:12,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:28:12,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:28:12,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14888.37 MB 2025-02-14 20:28:12,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:28:12,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:28:12,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:28:12,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.53 MB 2025-02-14 20:28:12,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15081.61 MB 2025-02-14 20:28:12,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 505.08 MB 2025-02-14 20:28:12,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:28:12,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:28:12,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:28:12,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16290.82 MB 2025-02-14 20:28:12,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:28:12,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:28:12,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:28:12,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14160.94 MB 2025-02-14 20:28:12,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15081.61 MB 2025-02-14 20:28:12,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 920.67 MB 2025-02-14 20:28:12,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:28:12,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19056.82 MB 2025-02-14 20:28:12,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:28:12,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16290.82 MB 2025-02-14 20:28:12,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:28:12,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:28:12,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 20:28:12,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15568.94 MB 2025-02-14 20:28:12,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15780.93 MB 2025-02-14 20:28:12,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.99 MB 2025-02-14 20:28:12,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19056.82 MB 2025-02-14 20:28:12,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19193.14 MB 2025-02-14 20:28:12,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-14 20:28:12,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15936.64 MB 2025-02-14 20:28:12,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:28:12,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:28:12,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:28:12,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15915.03 MB 2025-02-14 20:28:12,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16125.51 MB 2025-02-14 20:28:12,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.48 MB 2025-02-14 20:28:12,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19193.14 MB 2025-02-14 20:28:12,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19193.14 MB 2025-02-14 20:28:12,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:28:12,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16125.51 MB 2025-02-14 20:28:12,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:28:12,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:28:12,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 20:28:12,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13275.31 MB 2025-02-14 20:28:12,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16313.13 MB 2025-02-14 20:28:12,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3037.82 MB 2025-02-14 20:28:12,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43994.05 MB 2025-02-14 20:28:12,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19193.14 MB 2025-02-14 20:28:12,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24800.92 MB 2025-02-14 20:28:12,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16313.13 MB 2025-02-14 20:28:12,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:28:12,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:28:12,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 20:28:12,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13786.50 MB 2025-02-14 20:28:12,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16598.88 MB 2025-02-14 20:28:12,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2812.39 MB 2025-02-14 20:28:12,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19193.14 MB 2025-02-14 20:28:12,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19193.14 MB 2025-02-14 20:28:12,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:28:12,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16880.09 MB 2025-02-14 20:28:12,989 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7615, cut from 7617 2025-02-14 20:28:12,989 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:28:12,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:28:12,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:28:12,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:28:12,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:28:12,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16598.88 MB 2025-02-14 20:28:12,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24472.97 MB 2025-02-14 20:28:12,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7874.08 MB 2025-02-14 20:28:12,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19193.14 MB 2025-02-14 20:28:12,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28982.64 MB 2025-02-14 20:28:12,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9789.51 MB 2025-02-14 20:28:12,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24472.97 MB 2025-02-14 20:28:13,140 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7407] 2025-02-14 20:28:13,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:28:13,142 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:28:13,143 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:28:13,143 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:28:13,147 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:28:13,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:28:13,149 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:28:13,149 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:29:01,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:29:01,079 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:29:01,084 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:29:01,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:29:01,088 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 640, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:29:01,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:29:01,089 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 640, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:29:10,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:29:10,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:29:10,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.78 seconds 2025-02-14 20:29:10,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:10,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17428.33 MB 2025-02-14 20:29:10,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19693.25 MB 2025-02-14 20:29:10,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2264.92 MB 2025-02-14 20:29:10,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36813.41 MB 2025-02-14 20:29:10,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23985.13 MB 2025-02-14 20:29:10,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12828.28 MB 2025-02-14 20:29:10,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28711.64 MB 2025-02-14 20:29:10,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:29:10,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:29:10,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 20:29:10,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:10,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19693.25 MB 2025-02-14 20:29:10,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19105.01 MB 2025-02-14 20:29:10,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -588.24 MB 2025-02-14 20:29:10,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23985.13 MB 2025-02-14 20:29:10,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31033.66 MB 2025-02-14 20:29:10,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7048.53 MB 2025-02-14 20:29:10,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28133.34 MB 2025-02-14 20:29:12,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:29:12,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:29:12,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 20:29:12,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:12,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19105.01 MB 2025-02-14 20:29:12,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19635.85 MB 2025-02-14 20:29:12,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:29:12,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31033.66 MB 2025-02-14 20:29:12,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-14 20:29:12,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5632.95 MB 2025-02-14 20:29:12,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23614.40 MB 2025-02-14 20:29:12,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:29:12,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:29:12,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:29:12,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:12,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19635.85 MB 2025-02-14 20:29:12,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21525.39 MB 2025-02-14 20:29:12,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:29:12,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-14 20:29:12,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-14 20:29:12,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:29:12,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22942.82 MB 2025-02-14 20:29:13,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:29:13,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:29:13,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:29:13,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:13,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21525.39 MB 2025-02-14 20:29:13,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23767.24 MB 2025-02-14 20:29:13,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:29:13,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-14 20:29:13,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31065.11 MB 2025-02-14 20:29:13,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 20:29:13,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29311.52 MB 2025-02-14 20:29:13,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:29:13,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:29:13,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:29:13,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:13,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19635.85 MB 2025-02-14 20:29:13,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23767.24 MB 2025-02-14 20:29:13,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:29:13,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-14 20:29:13,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31065.11 MB 2025-02-14 20:29:13,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 20:29:13,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29311.52 MB 2025-02-14 20:29:13,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:29:13,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:29:13,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:29:13,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:13,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25300.78 MB 2025-02-14 20:29:13,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26067.79 MB 2025-02-14 20:29:13,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:29:13,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31065.11 MB 2025-02-14 20:29:13,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31482.45 MB 2025-02-14 20:29:13,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:29:13,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26775.58 MB 2025-02-14 20:29:13,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:29:13,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:29:13,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:29:13,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:13,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26480.68 MB 2025-02-14 20:29:13,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26710.16 MB 2025-02-14 20:29:13,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.48 MB 2025-02-14 20:29:13,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31482.45 MB 2025-02-14 20:29:13,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31482.45 MB 2025-02-14 20:29:13,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:29:13,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26901.18 MB 2025-02-14 20:29:13,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:29:13,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:29:13,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.14 seconds 2025-02-14 20:29:13,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:13,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15198.52 MB 2025-02-14 20:29:13,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26911.23 MB 2025-02-14 20:29:13,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11712.71 MB 2025-02-14 20:29:13,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36813.41 MB 2025-02-14 20:29:13,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31482.45 MB 2025-02-14 20:29:13,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5330.96 MB 2025-02-14 20:29:13,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26911.23 MB 2025-02-14 20:29:13,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:29:13,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:29:13,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:29:13,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:13,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26911.23 MB 2025-02-14 20:29:13,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20202.46 MB 2025-02-14 20:29:13,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6708.77 MB 2025-02-14 20:29:13,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31482.45 MB 2025-02-14 20:29:13,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31482.45 MB 2025-02-14 20:29:13,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:29:13,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29422.90 MB 2025-02-14 20:29:13,542 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:29:13,543 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:29:13,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:29:13,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:29:13,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 20:29:13,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:29:13,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20202.46 MB 2025-02-14 20:29:13,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28641.48 MB 2025-02-14 20:29:13,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:29:13,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31482.45 MB 2025-02-14 20:29:13,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39873.15 MB 2025-02-14 20:29:13,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:29:13,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28641.48 MB 2025-02-14 20:29:13,709 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:29:13,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:29:13,710 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:29:13,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:29:13,711 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:29:13,716 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:29:13,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:29:13,717 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:29:13,717 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:30:10,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:30:10,362 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:30:10,367 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:30:10,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:30:10,371 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:30:10,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:30:10,372 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:30:25,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:30:25,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:30:25,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.35 seconds 2025-02-14 20:30:25,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:25,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19964.74 MB 2025-02-14 20:30:25,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23517.84 MB 2025-02-14 20:30:25,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3553.10 MB 2025-02-14 20:30:25,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52458.16 MB 2025-02-14 20:30:25,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27541.90 MB 2025-02-14 20:30:25,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24916.26 MB 2025-02-14 20:30:25,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32380.51 MB 2025-02-14 20:30:25,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:30:25,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:30:25,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:30:25,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:25,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23517.84 MB 2025-02-14 20:30:25,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20997.33 MB 2025-02-14 20:30:25,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2520.51 MB 2025-02-14 20:30:25,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27541.90 MB 2025-02-14 20:30:25,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40816.87 MB 2025-02-14 20:30:25,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13274.97 MB 2025-02-14 20:30:25,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34836.50 MB 2025-02-14 20:30:27,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:30:27,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:30:27,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 20:30:27,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:27,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20997.33 MB 2025-02-14 20:30:27,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21528.18 MB 2025-02-14 20:30:27,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:30:27,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40816.87 MB 2025-02-14 20:30:27,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23444.06 MB 2025-02-14 20:30:27,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17372.81 MB 2025-02-14 20:30:27,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25507.76 MB 2025-02-14 20:30:27,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:30:27,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:30:27,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:30:27,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:27,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21528.18 MB 2025-02-14 20:30:27,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23417.71 MB 2025-02-14 20:30:27,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:30:27,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23444.06 MB 2025-02-14 20:30:27,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26747.08 MB 2025-02-14 20:30:27,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 20:30:27,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24835.14 MB 2025-02-14 20:30:27,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:30:27,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:30:27,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:30:27,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:27,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23417.71 MB 2025-02-14 20:30:27,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25659.57 MB 2025-02-14 20:30:27,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:30:27,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26747.08 MB 2025-02-14 20:30:27,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32881.25 MB 2025-02-14 20:30:27,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:30:27,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31203.85 MB 2025-02-14 20:30:27,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:30:27,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:30:27,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:30:27,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:27,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21528.18 MB 2025-02-14 20:30:27,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25659.57 MB 2025-02-14 20:30:27,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:30:27,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23444.06 MB 2025-02-14 20:30:27,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32881.25 MB 2025-02-14 20:30:27,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 20:30:27,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31203.85 MB 2025-02-14 20:30:28,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:30:28,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:30:28,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:30:28,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:28,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27193.11 MB 2025-02-14 20:30:28,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27960.11 MB 2025-02-14 20:30:28,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:30:28,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32881.25 MB 2025-02-14 20:30:28,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33298.58 MB 2025-02-14 20:30:28,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:30:28,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28667.90 MB 2025-02-14 20:30:28,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:30:28,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:30:28,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:30:28,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:28,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28373.00 MB 2025-02-14 20:30:28,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28601.04 MB 2025-02-14 20:30:28,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.04 MB 2025-02-14 20:30:28,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33298.58 MB 2025-02-14 20:30:28,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33298.58 MB 2025-02-14 20:30:28,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:30:28,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28804.01 MB 2025-02-14 20:30:28,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:30:28,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:30:28,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.76 seconds 2025-02-14 20:30:28,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:28,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16466.72 MB 2025-02-14 20:30:28,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28801.40 MB 2025-02-14 20:30:28,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12334.67 MB 2025-02-14 20:30:28,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52458.16 MB 2025-02-14 20:30:28,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33298.58 MB 2025-02-14 20:30:28,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19159.58 MB 2025-02-14 20:30:28,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28804.01 MB 2025-02-14 20:30:28,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:30:28,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:30:28,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:30:28,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:28,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28801.40 MB 2025-02-14 20:30:28,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21460.19 MB 2025-02-14 20:30:28,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7341.21 MB 2025-02-14 20:30:28,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33298.58 MB 2025-02-14 20:30:28,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33298.58 MB 2025-02-14 20:30:28,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:30:28,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31304.28 MB 2025-02-14 20:30:28,417 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 20:30:28,417 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:30:28,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:30:28,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:30:28,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:30:28,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:30:28,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21460.19 MB 2025-02-14 20:30:28,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29869.49 MB 2025-02-14 20:30:28,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 20:30:28,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33298.58 MB 2025-02-14 20:30:28,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41657.83 MB 2025-02-14 20:30:28,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 20:30:28,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29869.49 MB 2025-02-14 20:30:28,580 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 20:30:28,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:30:28,581 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:30:28,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:30:28,582 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:30:28,587 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:30:28,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:30:28,588 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:30:28,588 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:31:16,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:31:16,944 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:31:16,949 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:31:16,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:31:16,952 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:31:16,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:31:16,953 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:31:35,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:31:35,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:31:35,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.37 seconds 2025-02-14 20:31:35,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:35,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21309.59 MB 2025-02-14 20:31:35,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25545.84 MB 2025-02-14 20:31:35,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4236.25 MB 2025-02-14 20:31:35,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50017.08 MB 2025-02-14 20:31:35,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30446.45 MB 2025-02-14 20:31:35,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19570.62 MB 2025-02-14 20:31:35,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34404.84 MB 2025-02-14 20:31:35,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:31:35,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:31:35,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:31:35,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:35,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25545.84 MB 2025-02-14 20:31:35,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22001.73 MB 2025-02-14 20:31:35,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3544.11 MB 2025-02-14 20:31:35,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30446.45 MB 2025-02-14 20:31:35,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43859.84 MB 2025-02-14 20:31:35,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13413.38 MB 2025-02-14 20:31:35,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37782.97 MB 2025-02-14 20:31:37,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:31:37,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:31:37,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 20:31:37,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22001.73 MB 2025-02-14 20:31:37,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22532.57 MB 2025-02-14 20:31:37,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:31:37,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43859.84 MB 2025-02-14 20:31:37,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28334.62 MB 2025-02-14 20:31:37,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15525.22 MB 2025-02-14 20:31:37,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26511.12 MB 2025-02-14 20:31:37,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:31:37,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:31:37,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:31:37,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22532.57 MB 2025-02-14 20:31:37,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24422.10 MB 2025-02-14 20:31:37,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:31:37,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28334.62 MB 2025-02-14 20:31:37,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28334.62 MB 2025-02-14 20:31:37,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:31:37,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25839.53 MB 2025-02-14 20:31:37,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:31:37,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:31:37,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:31:37,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24422.10 MB 2025-02-14 20:31:37,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26663.96 MB 2025-02-14 20:31:37,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:31:37,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28334.62 MB 2025-02-14 20:31:37,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 20:31:37,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:31:37,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32208.24 MB 2025-02-14 20:31:37,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:31:37,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:31:37,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:31:37,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22532.57 MB 2025-02-14 20:31:37,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26663.96 MB 2025-02-14 20:31:37,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:31:37,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28334.62 MB 2025-02-14 20:31:37,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 20:31:37,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:31:37,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32208.24 MB 2025-02-14 20:31:37,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:31:37,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:31:37,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:31:37,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28197.50 MB 2025-02-14 20:31:37,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28964.50 MB 2025-02-14 20:31:37,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:31:37,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 20:31:37,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34884.03 MB 2025-02-14 20:31:37,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:31:37,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29672.29 MB 2025-02-14 20:31:37,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:31:37,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:31:37,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:31:37,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29377.39 MB 2025-02-14 20:31:37,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29606.43 MB 2025-02-14 20:31:37,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-14 20:31:37,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34884.03 MB 2025-02-14 20:31:37,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34884.03 MB 2025-02-14 20:31:37,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:31:37,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29820.19 MB 2025-02-14 20:31:37,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:31:37,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:31:37,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.76 seconds 2025-02-14 20:31:37,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17139.15 MB 2025-02-14 20:31:37,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29807.38 MB 2025-02-14 20:31:37,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12668.23 MB 2025-02-14 20:31:37,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50017.08 MB 2025-02-14 20:31:37,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34884.03 MB 2025-02-14 20:31:37,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15133.05 MB 2025-02-14 20:31:37,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29820.19 MB 2025-02-14 20:31:37,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:31:37,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:31:37,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:31:37,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:37,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29807.38 MB 2025-02-14 20:31:37,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22141.63 MB 2025-02-14 20:31:37,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7665.74 MB 2025-02-14 20:31:37,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34884.03 MB 2025-02-14 20:31:37,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34884.03 MB 2025-02-14 20:31:37,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:31:37,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32317.51 MB 2025-02-14 20:31:37,998 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 20:31:37,999 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:31:38,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:31:38,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:31:38,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:31:38,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:31:38,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22141.63 MB 2025-02-14 20:31:38,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30576.25 MB 2025-02-14 20:31:38,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 20:31:38,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34884.03 MB 2025-02-14 20:31:38,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45365.59 MB 2025-02-14 20:31:38,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 20:31:38,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30576.25 MB 2025-02-14 20:31:38,161 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 20:31:38,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:31:38,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:31:38,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:31:38,164 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:31:38,168 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:31:38,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:31:38,169 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:31:38,170 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:33:07,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:33:07,003 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:33:07,008 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:33:07,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:33:07,012 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1074, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:33:07,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:33:07,013 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1074, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:33:23,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:33:23,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:33:23,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.40 seconds 2025-02-14 20:33:23,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:23,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20452.51 MB 2025-02-14 20:33:23,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24253.34 MB 2025-02-14 20:33:23,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3800.83 MB 2025-02-14 20:33:23,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53750.01 MB 2025-02-14 20:33:23,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28307.36 MB 2025-02-14 20:33:23,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25442.65 MB 2025-02-14 20:33:23,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33094.78 MB 2025-02-14 20:33:23,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:33:23,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:33:23,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:33:23,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:23,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24253.34 MB 2025-02-14 20:33:23,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21362.29 MB 2025-02-14 20:33:23,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2891.05 MB 2025-02-14 20:33:23,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28307.36 MB 2025-02-14 20:33:23,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43029.36 MB 2025-02-14 20:33:23,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14722.01 MB 2025-02-14 20:33:23,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35917.26 MB 2025-02-14 20:33:25,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:33:25,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:33:25,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 20:33:25,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:25,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21362.29 MB 2025-02-14 20:33:25,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21893.13 MB 2025-02-14 20:33:25,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:33:25,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43029.36 MB 2025-02-14 20:33:25,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31847.35 MB 2025-02-14 20:33:25,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11182.01 MB 2025-02-14 20:33:25,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25871.68 MB 2025-02-14 20:33:25,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:33:25,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:33:25,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:33:25,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:25,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21893.13 MB 2025-02-14 20:33:25,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23782.67 MB 2025-02-14 20:33:25,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:33:25,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31847.35 MB 2025-02-14 20:33:25,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31847.35 MB 2025-02-14 20:33:25,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:33:25,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25200.10 MB 2025-02-14 20:33:25,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:33:25,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:33:25,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:33:25,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:25,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23782.67 MB 2025-02-14 20:33:25,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.52 MB 2025-02-14 20:33:25,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:33:25,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31847.35 MB 2025-02-14 20:33:25,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34206.65 MB 2025-02-14 20:33:25,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 20:33:25,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31568.80 MB 2025-02-14 20:33:25,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:33:25,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:33:25,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:33:25,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:25,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21893.13 MB 2025-02-14 20:33:25,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.52 MB 2025-02-14 20:33:25,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:33:25,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31847.35 MB 2025-02-14 20:33:25,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34206.65 MB 2025-02-14 20:33:25,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 20:33:25,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31568.80 MB 2025-02-14 20:33:25,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:33:25,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:33:25,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:33:25,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:25,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27558.06 MB 2025-02-14 20:33:25,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28325.07 MB 2025-02-14 20:33:25,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:33:25,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34206.65 MB 2025-02-14 20:33:25,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34621.88 MB 2025-02-14 20:33:25,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:33:25,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29032.86 MB 2025-02-14 20:33:25,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:33:25,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:33:25,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:33:25,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:25,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28737.96 MB 2025-02-14 20:33:25,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28966.33 MB 2025-02-14 20:33:25,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.37 MB 2025-02-14 20:33:25,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34621.88 MB 2025-02-14 20:33:25,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34621.88 MB 2025-02-14 20:33:25,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:33:25,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29170.40 MB 2025-02-14 20:33:25,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:33:25,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:33:25,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.79 seconds 2025-02-14 20:33:25,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:25,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16710.61 MB 2025-02-14 20:33:25,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29166.81 MB 2025-02-14 20:33:25,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12456.20 MB 2025-02-14 20:33:25,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53750.01 MB 2025-02-14 20:33:25,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34621.88 MB 2025-02-14 20:33:25,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19128.12 MB 2025-02-14 20:33:25,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29170.40 MB 2025-02-14 20:33:26,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:33:26,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:33:26,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:33:26,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:26,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29166.81 MB 2025-02-14 20:33:26,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21703.00 MB 2025-02-14 20:33:26,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7463.80 MB 2025-02-14 20:33:26,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34621.88 MB 2025-02-14 20:33:26,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34621.88 MB 2025-02-14 20:33:26,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:33:26,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31668.65 MB 2025-02-14 20:33:26,092 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-14 20:33:26,092 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:33:26,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:33:26,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:33:26,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:33:26,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:33:26,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21703.00 MB 2025-02-14 20:33:26,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30108.66 MB 2025-02-14 20:33:26,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-14 20:33:26,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34621.88 MB 2025-02-14 20:33:26,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42981.13 MB 2025-02-14 20:33:26,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 20:33:26,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30108.66 MB 2025-02-14 20:33:26,257 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-14 20:33:26,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:33:26,259 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:33:26,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:33:26,260 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:33:26,264 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:33:26,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:33:26,265 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:33:26,265 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:34:15,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:34:15,988 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:34:15,993 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:34:15,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:34:15,997 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:34:15,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:34:15,998 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:34:49,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:34:49,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:34:49,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.15 seconds 2025-02-14 20:34:49,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:49,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27978.12 MB 2025-02-14 20:34:49,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35601.27 MB 2025-02-14 20:34:49,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7623.15 MB 2025-02-14 20:34:49,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51340.38 MB 2025-02-14 20:34:49,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38015.07 MB 2025-02-14 20:34:49,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13325.30 MB 2025-02-14 20:34:49,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44471.57 MB 2025-02-14 20:34:49,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:34:49,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:34:49,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 20:34:49,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:49,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35601.27 MB 2025-02-14 20:34:49,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26976.87 MB 2025-02-14 20:34:49,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8624.40 MB 2025-02-14 20:34:49,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38015.07 MB 2025-02-14 20:34:49,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69719.82 MB 2025-02-14 20:34:49,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 31704.74 MB 2025-02-14 20:34:49,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57219.68 MB 2025-02-14 20:34:51,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:34:51,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:34:51,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 20:34:51,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26976.87 MB 2025-02-14 20:34:51,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27507.72 MB 2025-02-14 20:34:51,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:34:51,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69719.82 MB 2025-02-14 20:34:51,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29578.23 MB 2025-02-14 20:34:51,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40141.59 MB 2025-02-14 20:34:51,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31487.30 MB 2025-02-14 20:34:51,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:34:51,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:34:51,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:34:51,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27507.72 MB 2025-02-14 20:34:51,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29396.99 MB 2025-02-14 20:34:51,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-14 20:34:51,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29578.23 MB 2025-02-14 20:34:51,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32409.39 MB 2025-02-14 20:34:51,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 20:34:51,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30814.42 MB 2025-02-14 20:34:51,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:34:51,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:34:51,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:34:51,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29396.99 MB 2025-02-14 20:34:51,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31638.84 MB 2025-02-14 20:34:51,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:34:51,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32409.39 MB 2025-02-14 20:34:51,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39015.42 MB 2025-02-14 20:34:51,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 20:34:51,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37183.12 MB 2025-02-14 20:34:51,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:34:51,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:34:51,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:34:51,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27507.72 MB 2025-02-14 20:34:51,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31638.84 MB 2025-02-14 20:34:51,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-14 20:34:51,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29578.23 MB 2025-02-14 20:34:51,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39015.42 MB 2025-02-14 20:34:51,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 20:34:51,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37183.12 MB 2025-02-14 20:34:51,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:34:51,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:34:51,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:34:51,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33172.39 MB 2025-02-14 20:34:51,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33939.39 MB 2025-02-14 20:34:51,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:34:51,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39015.42 MB 2025-02-14 20:34:51,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39428.55 MB 2025-02-14 20:34:51,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 20:34:51,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34647.18 MB 2025-02-14 20:34:51,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:34:51,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:34:51,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:34:51,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34352.28 MB 2025-02-14 20:34:51,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34580.70 MB 2025-02-14 20:34:51,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.42 MB 2025-02-14 20:34:51,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39428.55 MB 2025-02-14 20:34:51,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39428.55 MB 2025-02-14 20:34:51,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:34:51,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34807.56 MB 2025-02-14 20:34:51,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:34:51,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:34:51,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.67 seconds 2025-02-14 20:34:51,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20473.41 MB 2025-02-14 20:34:51,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34781.03 MB 2025-02-14 20:34:51,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14307.62 MB 2025-02-14 20:34:51,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51340.38 MB 2025-02-14 20:34:51,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39428.55 MB 2025-02-14 20:34:51,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11911.82 MB 2025-02-14 20:34:51,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34807.56 MB 2025-02-14 20:34:51,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:34:51,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:34:51,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:34:51,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34781.03 MB 2025-02-14 20:34:51,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25466.52 MB 2025-02-14 20:34:51,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9314.51 MB 2025-02-14 20:34:51,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39428.55 MB 2025-02-14 20:34:51,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39428.55 MB 2025-02-14 20:34:51,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:34:51,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37283.48 MB 2025-02-14 20:34:51,951 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 20:34:51,951 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:34:51,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:34:51,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:34:51,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:34:51,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:34:51,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25466.52 MB 2025-02-14 20:34:51,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33875.82 MB 2025-02-14 20:34:51,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 20:34:51,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39428.55 MB 2025-02-14 20:34:51,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47787.80 MB 2025-02-14 20:34:51,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 20:34:51,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33875.82 MB 2025-02-14 20:34:52,113 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 20:34:52,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:34:52,115 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:34:52,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:34:52,116 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:34:52,120 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:34:52,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:34:52,121 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:34:52,121 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:35:40,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:35:40,327 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:35:40,332 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:35:40,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:35:40,337 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1023, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:35:40,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:35:40,338 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1023, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:35:56,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:35:56,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:35:56,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.76 seconds 2025-02-14 20:35:56,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:56,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20097.13 MB 2025-02-14 20:35:56,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23717.47 MB 2025-02-14 20:35:56,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3620.34 MB 2025-02-14 20:35:56,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56147.05 MB 2025-02-14 20:35:56,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26208.11 MB 2025-02-14 20:35:56,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29938.94 MB 2025-02-14 20:35:56,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32595.64 MB 2025-02-14 20:35:56,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:35:56,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:35:56,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:35:56,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:56,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23717.47 MB 2025-02-14 20:35:56,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21097.16 MB 2025-02-14 20:35:56,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2620.32 MB 2025-02-14 20:35:56,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26208.11 MB 2025-02-14 20:35:56,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41217.43 MB 2025-02-14 20:35:56,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15009.32 MB 2025-02-14 20:35:56,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34142.46 MB 2025-02-14 20:35:58,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:35:58,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:35:58,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:35:58,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21097.16 MB 2025-02-14 20:35:58,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21628.00 MB 2025-02-14 20:35:58,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:35:58,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41217.43 MB 2025-02-14 20:35:58,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29041.36 MB 2025-02-14 20:35:58,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12176.06 MB 2025-02-14 20:35:58,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25606.55 MB 2025-02-14 20:35:58,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:35:58,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:35:58,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:35:58,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21628.00 MB 2025-02-14 20:35:58,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23517.53 MB 2025-02-14 20:35:58,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:35:58,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-14 20:35:58,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29041.36 MB 2025-02-14 20:35:58,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:35:58,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24934.96 MB 2025-02-14 20:35:58,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:35:58,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:35:58,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:35:58,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23517.53 MB 2025-02-14 20:35:58,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25759.39 MB 2025-02-14 20:35:58,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:35:58,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-14 20:35:58,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33288.09 MB 2025-02-14 20:35:58,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 20:35:58,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31303.67 MB 2025-02-14 20:35:58,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:35:58,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:35:58,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 20:35:58,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21628.00 MB 2025-02-14 20:35:58,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25759.39 MB 2025-02-14 20:35:58,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:35:58,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-14 20:35:58,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33288.09 MB 2025-02-14 20:35:58,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 20:35:58,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31303.67 MB 2025-02-14 20:35:58,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:35:58,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:35:58,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 20:35:58,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27292.93 MB 2025-02-14 20:35:58,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28059.93 MB 2025-02-14 20:35:58,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:35:58,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33288.09 MB 2025-02-14 20:35:58,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 20:35:58,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:35:58,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28767.72 MB 2025-02-14 20:35:58,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:35:58,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:35:58,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:35:58,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28472.82 MB 2025-02-14 20:35:58,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28701.71 MB 2025-02-14 20:35:58,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 20:35:58,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33705.43 MB 2025-02-14 20:35:58,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 20:35:58,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:35:58,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28898.24 MB 2025-02-14 20:35:58,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:35:58,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:35:58,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.22 seconds 2025-02-14 20:35:58,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16532.92 MB 2025-02-14 20:35:58,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28902.51 MB 2025-02-14 20:35:58,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12369.59 MB 2025-02-14 20:35:58,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56147.05 MB 2025-02-14 20:35:58,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 20:35:58,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22441.62 MB 2025-02-14 20:35:58,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28902.51 MB 2025-02-14 20:35:58,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:35:58,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:35:58,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:35:58,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28902.51 MB 2025-02-14 20:35:58,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21533.12 MB 2025-02-14 20:35:58,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7369.39 MB 2025-02-14 20:35:58,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33705.43 MB 2025-02-14 20:35:58,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 20:35:58,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:35:58,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31410.80 MB 2025-02-14 20:35:58,849 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 20:35:58,850 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:35:58,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:35:58,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:35:58,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:35:58,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:35:58,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21533.12 MB 2025-02-14 20:35:58,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29960.46 MB 2025-02-14 20:35:58,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 20:35:58,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33705.43 MB 2025-02-14 20:35:58,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42085.65 MB 2025-02-14 20:35:58,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 20:35:58,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29960.46 MB 2025-02-14 20:35:59,016 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 20:35:59,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:35:59,017 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:35:59,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:35:59,018 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:35:59,023 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:35:59,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:35:59,027 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:35:59,027 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:37:12,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:37:12,420 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:37:12,427 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:37:12,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:37:12,434 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1201, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:37:12,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:37:12,435 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1201, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:37:30,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:37:30,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:37:30,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.52 seconds 2025-02-14 20:37:30,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:30,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21337.47 MB 2025-02-14 20:37:30,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25588.39 MB 2025-02-14 20:37:30,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-14 20:37:30,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50465.87 MB 2025-02-14 20:37:30,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34649.15 MB 2025-02-14 20:37:30,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15816.72 MB 2025-02-14 20:37:30,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34432.72 MB 2025-02-14 20:37:31,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:37:31,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:37:31,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:37:31,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:31,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25588.39 MB 2025-02-14 20:37:31,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22021.48 MB 2025-02-14 20:37:31,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3566.92 MB 2025-02-14 20:37:31,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34649.15 MB 2025-02-14 20:37:31,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40386.95 MB 2025-02-14 20:37:31,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5737.81 MB 2025-02-14 20:37:31,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35589.64 MB 2025-02-14 20:37:32,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:37:32,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:37:32,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 20:37:32,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:32,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22021.48 MB 2025-02-14 20:37:32,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22552.32 MB 2025-02-14 20:37:32,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:37:32,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40386.95 MB 2025-02-14 20:37:32,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31813.80 MB 2025-02-14 20:37:32,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8573.16 MB 2025-02-14 20:37:32,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26530.86 MB 2025-02-14 20:37:32,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:37:32,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:37:32,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:37:32,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:32,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22552.32 MB 2025-02-14 20:37:32,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24441.85 MB 2025-02-14 20:37:32,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:37:32,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31813.80 MB 2025-02-14 20:37:32,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31813.80 MB 2025-02-14 20:37:32,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:37:32,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25859.28 MB 2025-02-14 20:37:33,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:37:33,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:37:33,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:37:33,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:33,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24441.85 MB 2025-02-14 20:37:33,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26683.71 MB 2025-02-14 20:37:33,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:37:33,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31813.80 MB 2025-02-14 20:37:33,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 20:37:33,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 20:37:33,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32227.99 MB 2025-02-14 20:37:33,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:37:33,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:37:33,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:37:33,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:33,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22552.32 MB 2025-02-14 20:37:33,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26683.71 MB 2025-02-14 20:37:33,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:37:33,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31813.80 MB 2025-02-14 20:37:33,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 20:37:33,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 20:37:33,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32227.99 MB 2025-02-14 20:37:33,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:37:33,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:37:33,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:37:33,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:33,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28217.25 MB 2025-02-14 20:37:33,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28984.25 MB 2025-02-14 20:37:33,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:37:33,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 20:37:33,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35062.28 MB 2025-02-14 20:37:33,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:37:33,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29692.04 MB 2025-02-14 20:37:33,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:37:33,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:37:33,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:37:33,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:33,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29397.14 MB 2025-02-14 20:37:33,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29627.71 MB 2025-02-14 20:37:33,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.57 MB 2025-02-14 20:37:33,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35062.28 MB 2025-02-14 20:37:33,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35062.28 MB 2025-02-14 20:37:33,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:37:33,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29850.15 MB 2025-02-14 20:37:33,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:37:33,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:37:33,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.91 seconds 2025-02-14 20:37:33,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:33,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17153.09 MB 2025-02-14 20:37:33,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29828.78 MB 2025-02-14 20:37:33,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12675.70 MB 2025-02-14 20:37:33,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50465.87 MB 2025-02-14 20:37:33,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35062.28 MB 2025-02-14 20:37:33,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15403.58 MB 2025-02-14 20:37:33,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29850.15 MB 2025-02-14 20:37:33,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:37:33,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:37:33,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:37:33,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:33,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29828.78 MB 2025-02-14 20:37:33,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22157.48 MB 2025-02-14 20:37:33,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7671.31 MB 2025-02-14 20:37:33,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35062.28 MB 2025-02-14 20:37:33,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35062.28 MB 2025-02-14 20:37:33,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:37:33,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32340.45 MB 2025-02-14 20:37:33,632 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:37:33,632 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:37:33,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:37:33,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:37:33,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:37:33,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:37:33,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22157.48 MB 2025-02-14 20:37:33,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30596.50 MB 2025-02-14 20:37:33,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:37:33,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35062.28 MB 2025-02-14 20:37:33,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43452.99 MB 2025-02-14 20:37:33,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:37:33,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30596.50 MB 2025-02-14 20:37:33,797 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:37:33,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:37:33,799 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:37:33,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:37:33,800 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:37:33,804 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:37:33,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:37:33,806 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:37:33,806 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:38:27,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:38:27,945 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:38:27,953 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:38:27,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:38:27,960 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:38:27,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:38:27,962 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:38:53,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:38:53,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:38:53,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.49 seconds 2025-02-14 20:38:53,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:53,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24466.17 MB 2025-02-14 20:38:53,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30305.43 MB 2025-02-14 20:38:53,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5839.26 MB 2025-02-14 20:38:53,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56038.00 MB 2025-02-14 20:38:53,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36240.88 MB 2025-02-14 20:38:53,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19797.11 MB 2025-02-14 20:38:53,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39147.67 MB 2025-02-14 20:38:53,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:38:53,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:38:53,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 20:38:53,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:53,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30305.43 MB 2025-02-14 20:38:53,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24355.69 MB 2025-02-14 20:38:53,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5949.74 MB 2025-02-14 20:38:53,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36240.88 MB 2025-02-14 20:38:53,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56392.42 MB 2025-02-14 20:38:53,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20151.53 MB 2025-02-14 20:38:53,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47509.39 MB 2025-02-14 20:38:55,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:38:55,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:38:55,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 20:38:55,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:55,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24355.69 MB 2025-02-14 20:38:55,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24886.53 MB 2025-02-14 20:38:55,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:38:55,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56392.42 MB 2025-02-14 20:38:55,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27900.51 MB 2025-02-14 20:38:55,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28491.91 MB 2025-02-14 20:38:55,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28866.15 MB 2025-02-14 20:38:55,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:38:55,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:38:55,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:38:55,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:55,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24886.53 MB 2025-02-14 20:38:55,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26776.06 MB 2025-02-14 20:38:55,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:38:55,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27900.51 MB 2025-02-14 20:38:55,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30731.67 MB 2025-02-14 20:38:55,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 20:38:55,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28193.49 MB 2025-02-14 20:38:55,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:38:55,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:38:55,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:38:55,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:55,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26776.06 MB 2025-02-14 20:38:55,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.92 MB 2025-02-14 20:38:55,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:38:55,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30731.67 MB 2025-02-14 20:38:55,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36865.84 MB 2025-02-14 20:38:55,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:38:55,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34562.20 MB 2025-02-14 20:38:55,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:38:55,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:38:55,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:38:55,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:55,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24886.53 MB 2025-02-14 20:38:55,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.92 MB 2025-02-14 20:38:55,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:38:55,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27900.51 MB 2025-02-14 20:38:55,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36865.84 MB 2025-02-14 20:38:55,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 20:38:55,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34562.20 MB 2025-02-14 20:38:55,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:38:55,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:38:55,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:38:55,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:55,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30551.46 MB 2025-02-14 20:38:55,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31318.46 MB 2025-02-14 20:38:55,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:38:55,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36865.84 MB 2025-02-14 20:38:55,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37281.07 MB 2025-02-14 20:38:55,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:38:55,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32026.25 MB 2025-02-14 20:38:55,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:38:55,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:38:55,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:38:55,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:55,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31731.35 MB 2025-02-14 20:38:55,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31960.72 MB 2025-02-14 20:38:55,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.37 MB 2025-02-14 20:38:55,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37281.07 MB 2025-02-14 20:38:55,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37281.07 MB 2025-02-14 20:38:55,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:38:55,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32203.44 MB 2025-02-14 20:38:55,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:38:55,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:38:55,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.98 seconds 2025-02-14 20:38:55,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:55,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18717.44 MB 2025-02-14 20:38:55,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32161.64 MB 2025-02-14 20:38:55,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13444.20 MB 2025-02-14 20:38:55,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56038.00 MB 2025-02-14 20:38:55,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37281.07 MB 2025-02-14 20:38:55,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18756.93 MB 2025-02-14 20:38:55,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32203.44 MB 2025-02-14 20:38:56,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:38:56,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:38:56,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:38:56,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:56,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32161.64 MB 2025-02-14 20:38:56,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23719.54 MB 2025-02-14 20:38:56,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8442.10 MB 2025-02-14 20:38:56,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37281.07 MB 2025-02-14 20:38:56,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37281.07 MB 2025-02-14 20:38:56,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:38:56,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34671.47 MB 2025-02-14 20:38:56,233 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 20:38:56,233 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:38:56,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:38:56,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:38:56,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:38:56,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:38:56,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23719.54 MB 2025-02-14 20:38:56,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32152.84 MB 2025-02-14 20:38:56,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 20:38:56,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37281.07 MB 2025-02-14 20:38:56,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45665.48 MB 2025-02-14 20:38:56,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 20:38:56,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32152.84 MB 2025-02-14 20:38:56,396 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 20:38:56,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:38:56,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:38:56,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:38:56,398 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:38:56,403 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:38:56,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:38:56,404 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:38:56,404 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:40:09,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:40:09,028 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:40:09,035 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:40:09,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:40:09,041 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1201, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:40:09,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:40:09,043 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1201, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:40:27,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:40:27,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:40:27,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.49 seconds 2025-02-14 20:40:27,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:27,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21337.47 MB 2025-02-14 20:40:27,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25588.39 MB 2025-02-14 20:40:27,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-14 20:40:27,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54049.90 MB 2025-02-14 20:40:27,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30737.96 MB 2025-02-14 20:40:27,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23311.94 MB 2025-02-14 20:40:27,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34432.72 MB 2025-02-14 20:40:27,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:40:27,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:40:27,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:40:27,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:27,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25588.39 MB 2025-02-14 20:40:27,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22022.52 MB 2025-02-14 20:40:27,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3565.87 MB 2025-02-14 20:40:27,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30737.96 MB 2025-02-14 20:40:27,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43411.05 MB 2025-02-14 20:40:27,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12673.09 MB 2025-02-14 20:40:27,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38203.90 MB 2025-02-14 20:40:29,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:40:29,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:40:29,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 20:40:29,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:29,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22022.52 MB 2025-02-14 20:40:29,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22553.37 MB 2025-02-14 20:40:29,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:40:29,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43411.05 MB 2025-02-14 20:40:29,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27195.87 MB 2025-02-14 20:40:29,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16215.18 MB 2025-02-14 20:40:29,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26531.91 MB 2025-02-14 20:40:29,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:40:29,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:40:29,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:40:29,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:29,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22553.37 MB 2025-02-14 20:40:29,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24442.90 MB 2025-02-14 20:40:29,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:40:29,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27195.87 MB 2025-02-14 20:40:29,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28139.59 MB 2025-02-14 20:40:29,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 20:40:29,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25860.33 MB 2025-02-14 20:40:29,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:40:29,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:40:29,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:40:29,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:29,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24442.90 MB 2025-02-14 20:40:29,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26684.76 MB 2025-02-14 20:40:29,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:40:29,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28139.59 MB 2025-02-14 20:40:29,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34745.61 MB 2025-02-14 20:40:29,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 20:40:29,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32229.04 MB 2025-02-14 20:40:29,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:40:29,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:40:29,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:40:29,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:29,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22553.37 MB 2025-02-14 20:40:29,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26684.76 MB 2025-02-14 20:40:29,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:40:29,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27195.87 MB 2025-02-14 20:40:29,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34745.61 MB 2025-02-14 20:40:29,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 20:40:29,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32229.04 MB 2025-02-14 20:40:29,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:40:29,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:40:29,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:40:29,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:29,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28218.30 MB 2025-02-14 20:40:29,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28985.30 MB 2025-02-14 20:40:29,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:40:29,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34745.61 MB 2025-02-14 20:40:29,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35160.85 MB 2025-02-14 20:40:29,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:40:29,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29693.09 MB 2025-02-14 20:40:29,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:40:29,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:40:29,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:40:29,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:29,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29398.19 MB 2025-02-14 20:40:29,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29626.93 MB 2025-02-14 20:40:29,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-14 20:40:29,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35160.85 MB 2025-02-14 20:40:29,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35160.85 MB 2025-02-14 20:40:29,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:40:29,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29865.32 MB 2025-02-14 20:40:29,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:40:29,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:40:29,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.90 seconds 2025-02-14 20:40:29,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:29,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17153.09 MB 2025-02-14 20:40:29,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29827.58 MB 2025-02-14 20:40:29,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12674.50 MB 2025-02-14 20:40:29,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54049.90 MB 2025-02-14 20:40:29,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35160.85 MB 2025-02-14 20:40:29,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18889.05 MB 2025-02-14 20:40:29,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29865.32 MB 2025-02-14 20:40:30,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:40:30,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:40:30,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 20:40:30,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:30,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29827.58 MB 2025-02-14 20:40:30,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22151.00 MB 2025-02-14 20:40:30,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7676.58 MB 2025-02-14 20:40:30,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35160.85 MB 2025-02-14 20:40:30,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35160.85 MB 2025-02-14 20:40:30,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:40:30,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32334.03 MB 2025-02-14 20:40:30,232 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 20:40:30,232 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:40:30,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:40:30,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:40:30,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:40:30,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:40:30,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22151.00 MB 2025-02-14 20:40:30,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30572.96 MB 2025-02-14 20:40:30,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 20:40:30,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35160.85 MB 2025-02-14 20:40:30,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43532.68 MB 2025-02-14 20:40:30,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 20:40:30,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30572.96 MB 2025-02-14 20:40:30,395 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 20:40:30,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:40:30,396 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:40:30,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:40:30,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:40:30,402 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:40:30,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:40:30,403 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:40:30,403 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:41:46,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:41:46,599 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:41:46,604 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:41:46,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:41:46,608 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:41:46,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:41:46,609 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:42:12,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:42:12,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:42:12,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.82 seconds 2025-02-14 20:42:12,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:12,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24654.31 MB 2025-02-14 20:42:12,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30589.25 MB 2025-02-14 20:42:12,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5934.94 MB 2025-02-14 20:42:12,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51904.51 MB 2025-02-14 20:42:12,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36601.59 MB 2025-02-14 20:42:12,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15302.92 MB 2025-02-14 20:42:12,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39562.31 MB 2025-02-14 20:42:12,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:42:12,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:42:12,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 20:42:12,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:12,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30589.25 MB 2025-02-14 20:42:12,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24496.05 MB 2025-02-14 20:42:12,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6093.20 MB 2025-02-14 20:42:12,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36601.59 MB 2025-02-14 20:42:12,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56402.90 MB 2025-02-14 20:42:12,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19801.31 MB 2025-02-14 20:42:12,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47685.96 MB 2025-02-14 20:42:14,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:42:14,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:42:14,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:42:14,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:14,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24496.05 MB 2025-02-14 20:42:14,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25026.89 MB 2025-02-14 20:42:14,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:42:14,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56402.90 MB 2025-02-14 20:42:14,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32082.23 MB 2025-02-14 20:42:14,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24320.67 MB 2025-02-14 20:42:14,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29005.44 MB 2025-02-14 20:42:14,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:42:14,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:42:14,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:42:14,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:14,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25026.89 MB 2025-02-14 20:42:14,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26916.43 MB 2025-02-14 20:42:14,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:42:14,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32082.23 MB 2025-02-14 20:42:14,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32082.23 MB 2025-02-14 20:42:14,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:42:14,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28333.85 MB 2025-02-14 20:42:14,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:42:14,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:42:14,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:42:14,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:14,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26916.43 MB 2025-02-14 20:42:14,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29158.28 MB 2025-02-14 20:42:14,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:42:14,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32082.23 MB 2025-02-14 20:42:14,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36800.82 MB 2025-02-14 20:42:14,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:42:14,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34702.56 MB 2025-02-14 20:42:14,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:42:14,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:42:14,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:42:14,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:14,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25026.89 MB 2025-02-14 20:42:14,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29158.28 MB 2025-02-14 20:42:14,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:42:14,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32082.23 MB 2025-02-14 20:42:14,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36800.82 MB 2025-02-14 20:42:14,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 20:42:14,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34702.56 MB 2025-02-14 20:42:14,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:42:14,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:42:14,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:42:14,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:14,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30691.82 MB 2025-02-14 20:42:14,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31458.83 MB 2025-02-14 20:42:14,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:42:14,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36800.82 MB 2025-02-14 20:42:14,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 20:42:14,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:42:14,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32166.61 MB 2025-02-14 20:42:14,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:42:14,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:42:14,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:42:14,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:14,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31871.71 MB 2025-02-14 20:42:14,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32100.66 MB 2025-02-14 20:42:14,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.95 MB 2025-02-14 20:42:14,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37216.06 MB 2025-02-14 20:42:14,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 20:42:14,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:42:14,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32341.62 MB 2025-02-14 20:42:14,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:42:14,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:42:14,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.26 seconds 2025-02-14 20:42:14,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:14,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18811.51 MB 2025-02-14 20:42:14,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32301.20 MB 2025-02-14 20:42:14,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13489.69 MB 2025-02-14 20:42:14,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51904.51 MB 2025-02-14 20:42:14,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 20:42:14,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14688.45 MB 2025-02-14 20:42:14,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32341.62 MB 2025-02-14 20:42:15,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:42:15,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:42:15,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:42:15,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:15,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32301.20 MB 2025-02-14 20:42:15,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23807.52 MB 2025-02-14 20:42:15,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8493.68 MB 2025-02-14 20:42:15,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37216.06 MB 2025-02-14 20:42:15,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37216.06 MB 2025-02-14 20:42:15,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:42:15,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34806.10 MB 2025-02-14 20:42:15,161 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 20:42:15,161 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:42:15,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:42:15,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:42:15,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:42:15,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:42:15,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23807.52 MB 2025-02-14 20:42:15,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32224.12 MB 2025-02-14 20:42:15,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 20:42:15,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37216.06 MB 2025-02-14 20:42:15,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45583.70 MB 2025-02-14 20:42:15,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 20:42:15,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32224.12 MB 2025-02-14 20:42:15,322 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 20:42:15,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:42:15,324 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:42:15,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:42:15,325 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:42:15,329 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:42:15,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:42:15,330 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:42:15,330 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:44:31,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:44:31,924 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:44:31,929 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:44:31,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:44:31,934 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1757, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:44:31,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:44:31,935 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1757, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:44:58,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:44:58,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:44:58,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.95 seconds 2025-02-14 20:44:58,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:44:58,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25211.76 MB 2025-02-14 20:44:58,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31429.82 MB 2025-02-14 20:44:58,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6218.06 MB 2025-02-14 20:44:58,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53951.33 MB 2025-02-14 20:44:58,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36878.42 MB 2025-02-14 20:44:58,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17072.91 MB 2025-02-14 20:44:58,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40346.25 MB 2025-02-14 20:44:59,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:44:59,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:44:59,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 20:44:59,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:44:59,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31429.82 MB 2025-02-14 20:44:59,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24911.95 MB 2025-02-14 20:44:59,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6517.87 MB 2025-02-14 20:44:59,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36878.42 MB 2025-02-14 20:44:59,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56497.27 MB 2025-02-14 20:44:59,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19618.86 MB 2025-02-14 20:44:59,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47946.45 MB 2025-02-14 20:45:00,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:45:00,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:45:00,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 20:45:00,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:00,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24911.95 MB 2025-02-14 20:45:00,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25442.79 MB 2025-02-14 20:45:00,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:45:00,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56497.27 MB 2025-02-14 20:45:00,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27896.32 MB 2025-02-14 20:45:00,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28600.96 MB 2025-02-14 20:45:00,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29422.37 MB 2025-02-14 20:45:00,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:45:00,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:45:00,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:45:00,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:00,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25442.79 MB 2025-02-14 20:45:00,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27332.32 MB 2025-02-14 20:45:00,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:45:00,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27896.32 MB 2025-02-14 20:45:00,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30727.47 MB 2025-02-14 20:45:00,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 20:45:00,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28749.75 MB 2025-02-14 20:45:01,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:45:01,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:45:01,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:45:01,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:01,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27332.32 MB 2025-02-14 20:45:01,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29574.18 MB 2025-02-14 20:45:01,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:45:01,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30727.47 MB 2025-02-14 20:45:01,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36861.64 MB 2025-02-14 20:45:01,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:45:01,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35118.46 MB 2025-02-14 20:45:01,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:45:01,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:45:01,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:45:01,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:01,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25442.79 MB 2025-02-14 20:45:01,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29574.18 MB 2025-02-14 20:45:01,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:45:01,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27896.32 MB 2025-02-14 20:45:01,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36861.64 MB 2025-02-14 20:45:01,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 20:45:01,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35118.46 MB 2025-02-14 20:45:01,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:45:01,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:45:01,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:45:01,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:01,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31107.72 MB 2025-02-14 20:45:01,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31874.72 MB 2025-02-14 20:45:01,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:45:01,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36861.64 MB 2025-02-14 20:45:01,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37278.97 MB 2025-02-14 20:45:01,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:45:01,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32582.51 MB 2025-02-14 20:45:01,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:45:01,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:45:01,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:45:01,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:01,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32287.61 MB 2025-02-14 20:45:01,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32515.13 MB 2025-02-14 20:45:01,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.52 MB 2025-02-14 20:45:01,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37278.97 MB 2025-02-14 20:45:01,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37278.97 MB 2025-02-14 20:45:01,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:45:01,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32728.59 MB 2025-02-14 20:45:01,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:45:01,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:45:01,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.41 seconds 2025-02-14 20:45:01,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:01,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19090.23 MB 2025-02-14 20:45:01,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32715.12 MB 2025-02-14 20:45:01,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13624.89 MB 2025-02-14 20:45:01,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53951.33 MB 2025-02-14 20:45:01,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37278.97 MB 2025-02-14 20:45:01,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16672.36 MB 2025-02-14 20:45:01,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32728.59 MB 2025-02-14 20:45:01,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:45:01,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:45:01,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:45:01,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:01,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32715.12 MB 2025-02-14 20:45:01,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24078.35 MB 2025-02-14 20:45:01,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8636.77 MB 2025-02-14 20:45:01,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37278.97 MB 2025-02-14 20:45:01,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37278.97 MB 2025-02-14 20:45:01,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:45:01,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35213.28 MB 2025-02-14 20:45:01,634 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 20:45:01,635 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:45:01,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:45:01,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:45:01,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:45:01,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:45:01,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24078.35 MB 2025-02-14 20:45:01,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32471.63 MB 2025-02-14 20:45:01,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 20:45:01,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37278.97 MB 2025-02-14 20:45:01,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45625.64 MB 2025-02-14 20:45:01,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 20:45:01,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32471.63 MB 2025-02-14 20:45:01,798 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 20:45:01,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:45:01,800 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:45:01,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:45:01,800 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:45:01,805 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:45:01,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:45:01,806 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:45:01,806 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:45:22,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:45:22,384 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:45:22,389 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:45:22,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:45:22,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3305, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:45:22,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:45:22,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3305, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:46:13,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:46:13,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:46:13,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.29 seconds 2025-02-14 20:46:13,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:13,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36000.18 MB 2025-02-14 20:46:13,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47696.39 MB 2025-02-14 20:46:13,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11696.21 MB 2025-02-14 20:46:13,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77007.42 MB 2025-02-14 20:46:13,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52848.23 MB 2025-02-14 20:46:13,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24159.19 MB 2025-02-14 20:46:13,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59392.60 MB 2025-02-14 20:46:13,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:46:13,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:46:13,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:46:13,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:13,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47696.39 MB 2025-02-14 20:46:13,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32960.37 MB 2025-02-14 20:46:13,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14736.02 MB 2025-02-14 20:46:13,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52848.23 MB 2025-02-14 20:46:13,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 95328.14 MB 2025-02-14 20:46:13,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 42479.91 MB 2025-02-14 20:46:13,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 82003.83 MB 2025-02-14 20:46:15,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:46:15,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:46:15,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 20:46:15,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:15,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32960.37 MB 2025-02-14 20:46:15,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33491.21 MB 2025-02-14 20:46:15,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:46:15,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 95328.14 MB 2025-02-14 20:46:15,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36716.94 MB 2025-02-14 20:46:15,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -58611.20 MB 2025-02-14 20:46:15,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37470.79 MB 2025-02-14 20:46:15,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:46:15,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:46:15,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:46:15,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:15,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33491.21 MB 2025-02-14 20:46:15,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35380.47 MB 2025-02-14 20:46:15,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.26 MB 2025-02-14 20:46:15,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36716.94 MB 2025-02-14 20:46:15,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38604.37 MB 2025-02-14 20:46:15,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:46:15,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36797.90 MB 2025-02-14 20:46:16,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:46:16,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:46:16,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:46:16,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:16,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35380.47 MB 2025-02-14 20:46:16,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37622.32 MB 2025-02-14 20:46:16,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:46:16,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38604.37 MB 2025-02-14 20:46:16,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44738.54 MB 2025-02-14 20:46:16,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:46:16,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43166.60 MB 2025-02-14 20:46:16,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:46:16,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:46:16,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:46:16,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:16,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33491.21 MB 2025-02-14 20:46:16,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37622.32 MB 2025-02-14 20:46:16,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.12 MB 2025-02-14 20:46:16,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36716.94 MB 2025-02-14 20:46:16,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44738.54 MB 2025-02-14 20:46:16,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 20:46:16,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43166.60 MB 2025-02-14 20:46:16,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:46:16,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:46:16,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:46:16,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:16,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39155.87 MB 2025-02-14 20:46:16,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39922.87 MB 2025-02-14 20:46:16,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:46:16,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44738.54 MB 2025-02-14 20:46:16,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45153.78 MB 2025-02-14 20:46:16,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:46:16,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40630.66 MB 2025-02-14 20:46:16,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:46:16,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:46:16,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:46:16,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:16,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40335.76 MB 2025-02-14 20:46:16,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40564.15 MB 2025-02-14 20:46:16,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.39 MB 2025-02-14 20:46:16,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45153.78 MB 2025-02-14 20:46:16,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45153.78 MB 2025-02-14 20:46:16,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:46:16,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40784.99 MB 2025-02-14 20:46:16,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:46:16,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:46:16,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 53.91 seconds 2025-02-14 20:46:16,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:16,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24484.44 MB 2025-02-14 20:46:16,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40764.19 MB 2025-02-14 20:46:16,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16279.75 MB 2025-02-14 20:46:16,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65489.86 MB 2025-02-14 20:46:16,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45153.78 MB 2025-02-14 20:46:16,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20336.08 MB 2025-02-14 20:46:16,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40784.99 MB 2025-02-14 20:46:16,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:46:16,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:46:16,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:46:16,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:16,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40764.19 MB 2025-02-14 20:46:16,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29473.00 MB 2025-02-14 20:46:16,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11291.19 MB 2025-02-14 20:46:16,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45153.78 MB 2025-02-14 20:46:16,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45153.78 MB 2025-02-14 20:46:16,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:46:16,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43263.39 MB 2025-02-14 20:46:16,595 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 20:46:16,595 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:46:16,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:46:16,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:46:16,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:46:16,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:46:16,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29473.00 MB 2025-02-14 20:46:16,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37869.25 MB 2025-02-14 20:46:16,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.25 MB 2025-02-14 20:46:16,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45153.78 MB 2025-02-14 20:46:16,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49327.11 MB 2025-02-14 20:46:16,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 20:46:16,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37869.25 MB 2025-02-14 20:46:16,756 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 20:46:16,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:46:16,758 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:46:16,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:46:16,759 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:46:16,763 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:46:16,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:46:16,764 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:46:16,764 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:47:05,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:05,133 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:47:05,141 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:47:05,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:05,147 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 469, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:47:05,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:05,149 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 469, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:47:12,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:47:12,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:47:12,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.36 seconds 2025-02-14 20:47:12,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:12,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16236.77 MB 2025-02-14 20:47:12,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17896.54 MB 2025-02-14 20:47:12,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1659.76 MB 2025-02-14 20:47:12,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57673.78 MB 2025-02-14 20:47:12,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25161.63 MB 2025-02-14 20:47:12,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32512.15 MB 2025-02-14 20:47:12,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26840.61 MB 2025-02-14 20:47:12,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:47:12,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:47:12,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 20:47:12,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:12,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17896.54 MB 2025-02-14 20:47:12,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18217.08 MB 2025-02-14 20:47:12,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.55 MB 2025-02-14 20:47:12,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25161.63 MB 2025-02-14 20:47:12,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28701.62 MB 2025-02-14 20:47:12,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3539.99 MB 2025-02-14 20:47:12,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25124.85 MB 2025-02-14 20:47:14,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:47:14,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:47:14,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 20:47:14,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:14,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18217.08 MB 2025-02-14 20:47:14,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18747.93 MB 2025-02-14 20:47:14,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:47:14,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28701.62 MB 2025-02-14 20:47:14,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23066.57 MB 2025-02-14 20:47:14,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5635.05 MB 2025-02-14 20:47:14,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22726.47 MB 2025-02-14 20:47:14,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:47:14,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:47:14,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:47:14,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:14,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18747.93 MB 2025-02-14 20:47:14,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20637.46 MB 2025-02-14 20:47:14,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:47:14,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23066.57 MB 2025-02-14 20:47:14,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24010.29 MB 2025-02-14 20:47:14,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 20:47:14,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22054.89 MB 2025-02-14 20:47:14,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:47:14,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:47:14,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:47:14,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:14,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20637.46 MB 2025-02-14 20:47:14,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.32 MB 2025-02-14 20:47:14,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:47:14,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24010.29 MB 2025-02-14 20:47:14,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30146.56 MB 2025-02-14 20:47:14,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 20:47:14,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28423.60 MB 2025-02-14 20:47:14,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:47:14,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:47:14,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:47:14,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:14,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18747.93 MB 2025-02-14 20:47:14,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.32 MB 2025-02-14 20:47:14,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:47:14,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23066.57 MB 2025-02-14 20:47:14,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30146.56 MB 2025-02-14 20:47:14,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7079.99 MB 2025-02-14 20:47:14,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28423.60 MB 2025-02-14 20:47:14,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:47:14,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:47:14,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:47:14,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:14,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24412.86 MB 2025-02-14 20:47:14,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25179.86 MB 2025-02-14 20:47:14,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:47:14,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30146.56 MB 2025-02-14 20:47:14,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30563.89 MB 2025-02-14 20:47:14,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:47:14,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25887.65 MB 2025-02-14 20:47:14,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:47:14,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:47:14,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:47:14,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:14,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25592.75 MB 2025-02-14 20:47:14,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.11 MB 2025-02-14 20:47:14,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.36 MB 2025-02-14 20:47:14,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30563.89 MB 2025-02-14 20:47:14,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30563.89 MB 2025-02-14 20:47:14,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:14,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25984.26 MB 2025-02-14 20:47:14,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:47:14,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:47:14,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.72 seconds 2025-02-14 20:47:14,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:14,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14602.74 MB 2025-02-14 20:47:14,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26021.19 MB 2025-02-14 20:47:14,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11418.45 MB 2025-02-14 20:47:14,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57673.78 MB 2025-02-14 20:47:14,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30563.89 MB 2025-02-14 20:47:14,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27109.88 MB 2025-02-14 20:47:14,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26021.19 MB 2025-02-14 20:47:15,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:47:15,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:47:15,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:47:15,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:15,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26021.19 MB 2025-02-14 20:47:15,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19607.13 MB 2025-02-14 20:47:15,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6414.06 MB 2025-02-14 20:47:15,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30563.89 MB 2025-02-14 20:47:15,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30563.89 MB 2025-02-14 20:47:15,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:15,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28532.85 MB 2025-02-14 20:47:15,161 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:47:15,161 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:47:15,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:47:15,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:47:15,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:47:15,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:15,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19607.13 MB 2025-02-14 20:47:15,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28046.15 MB 2025-02-14 20:47:15,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:47:15,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30563.89 MB 2025-02-14 20:47:15,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38954.60 MB 2025-02-14 20:47:15,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:47:15,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28046.15 MB 2025-02-14 20:47:15,322 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:47:15,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:15,324 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:47:15,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:15,325 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:47:15,329 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:47:15,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:15,330 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:47:15,331 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:47:24,448 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:24,448 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:47:24,453 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:47:24,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:24,456 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 907, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:47:24,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:24,457 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 907, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:47:38,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:47:38,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:47:38,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.08 seconds 2025-02-14 20:47:38,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:38,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19288.83 MB 2025-02-14 20:47:38,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22499.57 MB 2025-02-14 20:47:38,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3210.74 MB 2025-02-14 20:47:38,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51539.61 MB 2025-02-14 20:47:38,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25516.05 MB 2025-02-14 20:47:38,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26023.56 MB 2025-02-14 20:47:38,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31478.91 MB 2025-02-14 20:47:38,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:47:38,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:47:38,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:47:38,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:38,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22499.57 MB 2025-02-14 20:47:38,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20494.11 MB 2025-02-14 20:47:38,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2005.46 MB 2025-02-14 20:47:38,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25516.05 MB 2025-02-14 20:47:38,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38344.33 MB 2025-02-14 20:47:38,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12828.28 MB 2025-02-14 20:47:38,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32762.91 MB 2025-02-14 20:47:40,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:47:40,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:47:40,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:47:40,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:40,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20494.11 MB 2025-02-14 20:47:40,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21024.95 MB 2025-02-14 20:47:40,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:47:40,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38344.33 MB 2025-02-14 20:47:40,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24429.72 MB 2025-02-14 20:47:40,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13914.60 MB 2025-02-14 20:47:40,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25003.50 MB 2025-02-14 20:47:40,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:47:40,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:47:40,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:47:40,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:40,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21024.95 MB 2025-02-14 20:47:40,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22914.49 MB 2025-02-14 20:47:40,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:47:40,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24429.72 MB 2025-02-14 20:47:40,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26319.26 MB 2025-02-14 20:47:40,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-14 20:47:40,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24331.91 MB 2025-02-14 20:47:40,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:47:40,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:47:40,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:47:40,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:40,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22914.49 MB 2025-02-14 20:47:40,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25156.34 MB 2025-02-14 20:47:40,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:47:40,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26319.26 MB 2025-02-14 20:47:40,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32453.43 MB 2025-02-14 20:47:40,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:47:40,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30700.62 MB 2025-02-14 20:47:40,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:47:40,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:47:40,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:47:40,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:40,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21024.95 MB 2025-02-14 20:47:40,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25156.34 MB 2025-02-14 20:47:40,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:47:40,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24429.72 MB 2025-02-14 20:47:40,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32453.43 MB 2025-02-14 20:47:40,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-14 20:47:40,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30700.62 MB 2025-02-14 20:47:40,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:47:40,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:47:40,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:47:40,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:40,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26689.88 MB 2025-02-14 20:47:40,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27456.89 MB 2025-02-14 20:47:40,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:47:40,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32453.43 MB 2025-02-14 20:47:40,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32870.76 MB 2025-02-14 20:47:40,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:47:40,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28164.67 MB 2025-02-14 20:47:40,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:47:40,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:47:40,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:47:40,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:40,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27869.77 MB 2025-02-14 20:47:40,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28099.31 MB 2025-02-14 20:47:40,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.54 MB 2025-02-14 20:47:40,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32870.76 MB 2025-02-14 20:47:40,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32870.76 MB 2025-02-14 20:47:40,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:40,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28316.92 MB 2025-02-14 20:47:40,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:47:40,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:47:40,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.47 seconds 2025-02-14 20:47:40,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:40,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16128.77 MB 2025-02-14 20:47:40,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28300.39 MB 2025-02-14 20:47:40,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12171.62 MB 2025-02-14 20:47:40,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51539.61 MB 2025-02-14 20:47:40,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32870.76 MB 2025-02-14 20:47:40,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18668.85 MB 2025-02-14 20:47:40,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28316.92 MB 2025-02-14 20:47:41,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:47:41,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:47:41,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:47:41,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:41,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28300.39 MB 2025-02-14 20:47:41,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21133.16 MB 2025-02-14 20:47:41,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7167.23 MB 2025-02-14 20:47:41,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32870.76 MB 2025-02-14 20:47:41,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32870.76 MB 2025-02-14 20:47:41,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:41,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30812.05 MB 2025-02-14 20:47:41,214 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:47:41,214 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:47:41,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:47:41,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:47:41,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:47:41,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:41,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21133.16 MB 2025-02-14 20:47:41,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29572.18 MB 2025-02-14 20:47:41,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:47:41,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32870.76 MB 2025-02-14 20:47:41,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43360.71 MB 2025-02-14 20:47:41,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 20:47:41,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29572.18 MB 2025-02-14 20:47:41,378 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:47:41,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:41,379 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:47:41,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:41,380 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:47:41,385 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:47:41,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:41,386 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:47:41,386 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:47:47,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:47,189 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:47:47,197 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:47:47,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:47,204 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:47:47,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:47,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:47:49,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:47:49,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:47:49,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.47 seconds 2025-02-14 20:47:49,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:49,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-14 20:47:49,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-14 20:47:49,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-14 20:47:49,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55945.72 MB 2025-02-14 20:47:49,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 20:47:49,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36417.04 MB 2025-02-14 20:47:49,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23499.24 MB 2025-02-14 20:47:49,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:47:49,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:47:49,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:47:49,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:49,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-14 20:47:49,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14699.99 MB 2025-02-14 20:47:49,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.21 MB 2025-02-14 20:47:49,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 20:47:49,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 20:47:49,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:49,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16476.33 MB 2025-02-14 20:47:50,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:47:50,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:47:50,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.68 seconds 2025-02-14 20:47:50,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14699.99 MB 2025-02-14 20:47:50,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14877.83 MB 2025-02-14 20:47:50,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-14 20:47:50,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 20:47:50,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 20:47:50,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:50,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18869.64 MB 2025-02-14 20:47:50,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:47:50,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:47:50,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:47:50,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.76 MB 2025-02-14 20:47:50,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15510.60 MB 2025-02-14 20:47:50,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-14 20:47:50,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 20:47:50,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 20:47:50,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:50,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15985.44 MB 2025-02-14 20:47:50,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:47:50,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:47:50,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:47:50,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15510.60 MB 2025-02-14 20:47:50,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16261.66 MB 2025-02-14 20:47:50,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-14 20:47:50,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 20:47:50,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 20:47:50,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:50,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18118.96 MB 2025-02-14 20:47:50,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:47:50,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:47:50,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 20:47:50,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.76 MB 2025-02-14 20:47:50,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16261.66 MB 2025-02-14 20:47:50,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-14 20:47:50,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 20:47:50,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 20:47:50,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:50,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18118.96 MB 2025-02-14 20:47:50,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:47:50,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:47:50,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:47:50,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16775.40 MB 2025-02-14 20:47:50,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17032.35 MB 2025-02-14 20:47:50,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-14 20:47:50,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 20:47:50,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-14 20:47:50,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 138.41 MB 2025-02-14 20:47:50,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17280.56 MB 2025-02-14 20:47:50,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:47:50,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:47:50,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:47:50,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17170.67 MB 2025-02-14 20:47:50,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17390.16 MB 2025-02-14 20:47:50,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.49 MB 2025-02-14 20:47:50,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19667.09 MB 2025-02-14 20:47:50,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-14 20:47:50,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:50,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17390.40 MB 2025-02-14 20:47:50,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:47:50,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:47:50,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-14 20:47:50,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-14 20:47:50,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14233.26 MB 2025-02-14 20:47:50,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.97 MB 2025-02-14 20:47:50,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55945.72 MB 2025-02-14 20:47:50,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-14 20:47:50,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36278.63 MB 2025-02-14 20:47:50,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17591.14 MB 2025-02-14 20:47:50,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:47:50,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:47:50,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 20:47:50,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14233.26 MB 2025-02-14 20:47:50,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17245.82 MB 2025-02-14 20:47:50,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.56 MB 2025-02-14 20:47:50,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19667.09 MB 2025-02-14 20:47:50,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-14 20:47:50,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:47:50,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17547.04 MB 2025-02-14 20:47:50,909 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 20:47:50,910 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:47:50,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:47:50,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:47:50,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:47:50,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:47:50,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17245.82 MB 2025-02-14 20:47:50,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25680.67 MB 2025-02-14 20:47:50,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 20:47:50,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19667.09 MB 2025-02-14 20:47:50,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30150.75 MB 2025-02-14 20:47:50,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10483.66 MB 2025-02-14 20:47:50,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25680.67 MB 2025-02-14 20:47:51,166 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 20:47:51,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:51,169 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:47:51,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:51,171 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:47:51,178 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:47:51,181 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:47:51,181 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:47:51,181 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:48:46,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:46,425 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:48:46,431 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 295 2025-02-14 20:48:46,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:46,435 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 116, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:48:46,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:46,436 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 116, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:48:48,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:48:48,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:48:48,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.79 seconds 2025-02-14 20:48:48,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:48,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19181.75 MB 2025-02-14 20:48:48,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19592.27 MB 2025-02-14 20:48:48,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.52 MB 2025-02-14 20:48:48,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42729.47 MB 2025-02-14 20:48:48,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23722.98 MB 2025-02-14 20:48:48,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19006.49 MB 2025-02-14 20:48:48,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28427.13 MB 2025-02-14 20:48:48,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:48:48,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:48:48,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:48:48,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:48,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19592.27 MB 2025-02-14 20:48:48,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19791.16 MB 2025-02-14 20:48:48,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 198.89 MB 2025-02-14 20:48:48,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 20:48:48,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23722.98 MB 2025-02-14 20:48:48,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:48,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20407.00 MB 2025-02-14 20:48:48,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:48:48,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:48:48,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.56 seconds 2025-02-14 20:48:48,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:48,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19791.16 MB 2025-02-14 20:48:48,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19945.11 MB 2025-02-14 20:48:48,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 153.94 MB 2025-02-14 20:48:48,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 20:48:48,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23722.98 MB 2025-02-14 20:48:48,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:48,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23876.92 MB 2025-02-14 20:48:48,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:48:48,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:48:48,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:48:48,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:48,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14540.30 MB 2025-02-14 20:48:48,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15088.14 MB 2025-02-14 20:48:48,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 547.83 MB 2025-02-14 20:48:48,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 20:48:48,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23722.98 MB 2025-02-14 20:48:48,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:48,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15499.20 MB 2025-02-14 20:48:48,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:48:48,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:48:48,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 20:48:48,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:48,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15088.14 MB 2025-02-14 20:48:48,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15753.52 MB 2025-02-14 20:48:48,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 665.39 MB 2025-02-14 20:48:48,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 20:48:48,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23722.98 MB 2025-02-14 20:48:48,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:48,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.12 MB 2025-02-14 20:48:48,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:48:48,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:48:48,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 20:48:48,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:48,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14540.30 MB 2025-02-14 20:48:48,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15753.52 MB 2025-02-14 20:48:48,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1213.22 MB 2025-02-14 20:48:48,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 20:48:48,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23722.98 MB 2025-02-14 20:48:48,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:48,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.12 MB 2025-02-14 20:48:49,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:48:49,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:48:49,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:48:49,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:49,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16395.91 MB 2025-02-14 20:48:49,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17140.36 MB 2025-02-14 20:48:49,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 744.45 MB 2025-02-14 20:48:49,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 20:48:49,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23842.52 MB 2025-02-14 20:48:49,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 119.54 MB 2025-02-14 20:48:49,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17345.62 MB 2025-02-14 20:48:49,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:48:49,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:48:49,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:48:49,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:49,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17465.36 MB 2025-02-14 20:48:49,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17558.54 MB 2025-02-14 20:48:49,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 93.18 MB 2025-02-14 20:48:49,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23842.52 MB 2025-02-14 20:48:49,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23842.52 MB 2025-02-14 20:48:49,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:49,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17558.54 MB 2025-02-14 20:48:49,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:48:49,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:48:49,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.60 seconds 2025-02-14 20:48:49,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:49,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18777.60 MB 2025-02-14 20:48:49,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17643.21 MB 2025-02-14 20:48:49,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1134.39 MB 2025-02-14 20:48:49,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42729.47 MB 2025-02-14 20:48:49,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23842.52 MB 2025-02-14 20:48:49,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18886.95 MB 2025-02-14 20:48:49,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17643.21 MB 2025-02-14 20:48:49,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:48:49,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:48:49,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 20:48:49,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:49,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14139.41 MB 2025-02-14 20:48:49,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15408.65 MB 2025-02-14 20:48:49,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1269.24 MB 2025-02-14 20:48:49,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23842.52 MB 2025-02-14 20:48:49,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23842.52 MB 2025-02-14 20:48:49,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:49,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15535.56 MB 2025-02-14 20:48:49,190 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 3429, cut from 3431 2025-02-14 20:48:49,191 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:48:49,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:48:49,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:48:49,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:48:49,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:48:49,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15408.65 MB 2025-02-14 20:48:49,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18961.97 MB 2025-02-14 20:48:49,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3553.33 MB 2025-02-14 20:48:49,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23842.52 MB 2025-02-14 20:48:49,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23842.52 MB 2025-02-14 20:48:49,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:48:49,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18961.97 MB 2025-02-14 20:48:49,306 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 3150] 2025-02-14 20:48:49,309 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:49,309 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 308, 128256]), torch.float32, cuda:0] 2025-02-14 20:48:49,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:49,311 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 309]), torch.int64, cuda:0] 2025-02-14 20:48:49,321 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [296, 308] 2025-02-14 20:48:49,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:49,323 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:48:49,323 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 20:48:57,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:57,752 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:48:57,757 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:48:57,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:57,760 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:48:57,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:48:57,761 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:49:17,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:49:17,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:49:17,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.00 seconds 2025-02-14 20:49:17,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:17,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22035.54 MB 2025-02-14 20:49:17,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26639.70 MB 2025-02-14 20:49:17,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4604.17 MB 2025-02-14 20:49:17,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30144.46 MB 2025-02-14 20:49:17,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31446.79 MB 2025-02-14 20:49:17,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1302.33 MB 2025-02-14 20:49:17,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35583.77 MB 2025-02-14 20:49:17,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:49:17,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:49:17,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:49:17,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:17,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26639.70 MB 2025-02-14 20:49:17,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22543.06 MB 2025-02-14 20:49:17,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4096.65 MB 2025-02-14 20:49:17,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31446.79 MB 2025-02-14 20:49:17,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47171.24 MB 2025-02-14 20:49:17,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15724.45 MB 2025-02-14 20:49:17,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40273.85 MB 2025-02-14 20:49:19,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:49:19,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:49:19,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 20:49:19,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:19,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22543.06 MB 2025-02-14 20:49:19,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23073.90 MB 2025-02-14 20:49:19,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:49:19,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47171.24 MB 2025-02-14 20:49:19,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26663.19 MB 2025-02-14 20:49:19,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20508.05 MB 2025-02-14 20:49:19,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27053.48 MB 2025-02-14 20:49:19,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:49:19,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:49:19,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:49:19,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:19,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23073.90 MB 2025-02-14 20:49:19,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24963.43 MB 2025-02-14 20:49:19,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:49:19,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26663.19 MB 2025-02-14 20:49:19,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27606.91 MB 2025-02-14 20:49:19,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 20:49:19,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26380.86 MB 2025-02-14 20:49:20,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:49:20,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:49:20,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:49:20,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:20,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24963.43 MB 2025-02-14 20:49:20,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27205.29 MB 2025-02-14 20:49:20,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:49:20,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27606.91 MB 2025-02-14 20:49:20,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34212.94 MB 2025-02-14 20:49:20,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 20:49:20,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32749.57 MB 2025-02-14 20:49:20,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:49:20,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:49:20,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:49:20,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:20,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23073.90 MB 2025-02-14 20:49:20,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27205.29 MB 2025-02-14 20:49:20,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:49:20,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26663.19 MB 2025-02-14 20:49:20,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34212.94 MB 2025-02-14 20:49:20,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 20:49:20,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32749.57 MB 2025-02-14 20:49:20,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:49:20,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:49:20,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:49:20,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:20,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28738.83 MB 2025-02-14 20:49:20,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29505.83 MB 2025-02-14 20:49:20,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:49:20,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34212.94 MB 2025-02-14 20:49:20,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34630.27 MB 2025-02-14 20:49:20,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:49:20,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30213.62 MB 2025-02-14 20:49:20,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:49:20,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:49:20,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:49:20,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:20,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29918.72 MB 2025-02-14 20:49:20,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30145.40 MB 2025-02-14 20:49:20,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.67 MB 2025-02-14 20:49:20,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34630.27 MB 2025-02-14 20:49:20,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34630.27 MB 2025-02-14 20:49:20,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:49:20,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30379.33 MB 2025-02-14 20:49:20,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:49:20,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:49:20,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.47 seconds 2025-02-14 20:49:20,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:20,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17501.49 MB 2025-02-14 20:49:20,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30345.88 MB 2025-02-14 20:49:20,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12844.38 MB 2025-02-14 20:49:20,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25610.42 MB 2025-02-14 20:49:20,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34630.27 MB 2025-02-14 20:49:20,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9019.85 MB 2025-02-14 20:49:20,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30379.33 MB 2025-02-14 20:49:20,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:49:20,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:49:20,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:49:20,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:20,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30345.88 MB 2025-02-14 20:49:20,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22487.83 MB 2025-02-14 20:49:20,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7858.05 MB 2025-02-14 20:49:20,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34630.27 MB 2025-02-14 20:49:20,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34630.27 MB 2025-02-14 20:49:20,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:49:20,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32842.49 MB 2025-02-14 20:49:20,523 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 20:49:20,524 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:49:20,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:49:20,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:49:20,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:49:20,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:49:20,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22487.83 MB 2025-02-14 20:49:20,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30876.25 MB 2025-02-14 20:49:20,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 20:49:20,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34630.27 MB 2025-02-14 20:49:20,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42970.64 MB 2025-02-14 20:49:20,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-14 20:49:20,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30876.25 MB 2025-02-14 20:49:20,689 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 20:49:20,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:49:20,691 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:49:20,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:49:20,692 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:49:20,696 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:49:20,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:49:20,697 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:49:20,697 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:50:39,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:39,765 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:50:39,770 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:50:39,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:39,775 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:50:39,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:39,776 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:50:42,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:50:42,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:50:42,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.72 seconds 2025-02-14 20:50:42,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:42,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14202.07 MB 2025-02-14 20:50:42,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14828.46 MB 2025-02-14 20:50:42,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.39 MB 2025-02-14 20:50:42,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55480.16 MB 2025-02-14 20:50:42,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 20:50:42,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35714.50 MB 2025-02-14 20:50:42,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23673.44 MB 2025-02-14 20:50:42,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:50:42,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:50:42,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:50:42,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:42,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14828.46 MB 2025-02-14 20:50:42,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15131.95 MB 2025-02-14 20:50:42,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 303.49 MB 2025-02-14 20:50:42,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 20:50:42,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 20:50:42,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:50:42,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17318.60 MB 2025-02-14 20:50:43,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:50:43,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:50:43,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 20:50:43,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15131.95 MB 2025-02-14 20:50:43,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15366.85 MB 2025-02-14 20:50:43,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 20:50:43,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 20:50:43,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20168.31 MB 2025-02-14 20:50:43,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 20:50:43,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19301.60 MB 2025-02-14 20:50:43,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:50:43,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:50:43,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:50:43,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15366.78 MB 2025-02-14 20:50:43,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16202.70 MB 2025-02-14 20:50:43,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 20:50:43,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20168.31 MB 2025-02-14 20:50:43,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20168.31 MB 2025-02-14 20:50:43,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:50:43,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16829.92 MB 2025-02-14 20:50:43,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:50:43,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:50:43,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 20:50:43,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16202.70 MB 2025-02-14 20:50:43,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17194.76 MB 2025-02-14 20:50:43,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 20:50:43,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20168.31 MB 2025-02-14 20:50:43,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21426.60 MB 2025-02-14 20:50:43,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-14 20:50:43,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19649.90 MB 2025-02-14 20:50:43,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:50:43,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:50:43,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 20:50:43,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15366.78 MB 2025-02-14 20:50:43,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17194.76 MB 2025-02-14 20:50:43,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 20:50:43,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20168.31 MB 2025-02-14 20:50:43,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21426.60 MB 2025-02-14 20:50:43,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-14 20:50:43,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19649.90 MB 2025-02-14 20:50:43,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:50:43,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:50:43,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:50:43,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17873.35 MB 2025-02-14 20:50:43,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18214.58 MB 2025-02-14 20:50:43,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.23 MB 2025-02-14 20:50:43,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21426.60 MB 2025-02-14 20:50:43,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21609.05 MB 2025-02-14 20:50:43,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 20:50:43,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18534.88 MB 2025-02-14 20:50:43,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:50:43,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:50:43,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:50:43,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18397.29 MB 2025-02-14 20:50:43,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18626.49 MB 2025-02-14 20:50:43,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.19 MB 2025-02-14 20:50:43,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21609.05 MB 2025-02-14 20:50:43,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21609.05 MB 2025-02-14 20:50:43,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:50:43,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18654.18 MB 2025-02-14 20:50:43,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:50:43,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:50:43,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.79 seconds 2025-02-14 20:50:43,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13585.39 MB 2025-02-14 20:50:43,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18827.56 MB 2025-02-14 20:50:43,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5242.17 MB 2025-02-14 20:50:43,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55480.16 MB 2025-02-14 20:50:43,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21609.05 MB 2025-02-14 20:50:43,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33871.10 MB 2025-02-14 20:50:43,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18827.56 MB 2025-02-14 20:50:43,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:50:43,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:50:43,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:50:43,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18827.56 MB 2025-02-14 20:50:43,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17539.96 MB 2025-02-14 20:50:43,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1287.60 MB 2025-02-14 20:50:43,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21609.05 MB 2025-02-14 20:50:43,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21609.05 MB 2025-02-14 20:50:43,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:50:43,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19062.88 MB 2025-02-14 20:50:43,862 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:50:43,863 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 20:50:43,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:50:43,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:50:43,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:50:43,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:50:43,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17539.96 MB 2025-02-14 20:50:43,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25978.99 MB 2025-02-14 20:50:43,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:50:43,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21609.05 MB 2025-02-14 20:50:43,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29999.76 MB 2025-02-14 20:50:43,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 20:50:43,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25978.99 MB 2025-02-14 20:50:44,038 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:50:44,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:44,039 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:50:44,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:44,040 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:50:44,045 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:50:44,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:44,046 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:50:44,046 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 20:50:52,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:52,038 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:50:52,043 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:50:52,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:52,046 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1973, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:50:52,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:50:52,047 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1973, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:51:22,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:51:22,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:51:22,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.46 seconds 2025-02-14 20:51:22,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:22,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26716.89 MB 2025-02-14 20:51:22,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33699.22 MB 2025-02-14 20:51:22,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6982.34 MB 2025-02-14 20:51:22,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42584.77 MB 2025-02-14 20:51:22,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37681.63 MB 2025-02-14 20:51:22,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4903.14 MB 2025-02-14 20:51:22,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42530.85 MB 2025-02-14 20:51:22,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:51:22,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:51:22,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 20:51:22,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:22,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33699.22 MB 2025-02-14 20:51:22,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26034.86 MB 2025-02-14 20:51:22,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7664.36 MB 2025-02-14 20:51:22,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37681.63 MB 2025-02-14 20:51:22,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64359.50 MB 2025-02-14 20:51:22,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 26677.87 MB 2025-02-14 20:51:22,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54272.42 MB 2025-02-14 20:51:24,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:51:24,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:51:24,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 20:51:24,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:24,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26034.86 MB 2025-02-14 20:51:24,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26565.70 MB 2025-02-14 20:51:24,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:51:24,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64359.50 MB 2025-02-14 20:51:24,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32113.69 MB 2025-02-14 20:51:24,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32245.81 MB 2025-02-14 20:51:24,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30544.25 MB 2025-02-14 20:51:24,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:51:24,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:51:24,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:51:24,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:24,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.70 MB 2025-02-14 20:51:24,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28455.24 MB 2025-02-14 20:51:24,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:51:24,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32113.69 MB 2025-02-14 20:51:24,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32113.69 MB 2025-02-14 20:51:24,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:51:24,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29872.67 MB 2025-02-14 20:51:24,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:51:24,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:51:24,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:51:24,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:24,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28455.24 MB 2025-02-14 20:51:24,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30697.09 MB 2025-02-14 20:51:24,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:51:24,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32113.69 MB 2025-02-14 20:51:24,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38484.84 MB 2025-02-14 20:51:24,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-14 20:51:24,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36241.38 MB 2025-02-14 20:51:24,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:51:24,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:51:24,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:51:24,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:24,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.70 MB 2025-02-14 20:51:24,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30697.09 MB 2025-02-14 20:51:24,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:51:24,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32113.69 MB 2025-02-14 20:51:24,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38484.84 MB 2025-02-14 20:51:24,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-14 20:51:24,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36241.38 MB 2025-02-14 20:51:24,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:51:24,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:51:24,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 20:51:24,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:24,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32230.64 MB 2025-02-14 20:51:24,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32997.64 MB 2025-02-14 20:51:24,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:51:24,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38484.84 MB 2025-02-14 20:51:24,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38900.07 MB 2025-02-14 20:51:24,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:51:24,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33705.43 MB 2025-02-14 20:51:25,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:51:25,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:51:25,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:51:25,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:25,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33410.53 MB 2025-02-14 20:51:25,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33638.70 MB 2025-02-14 20:51:25,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 20:51:25,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38900.07 MB 2025-02-14 20:51:25,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38900.07 MB 2025-02-14 20:51:25,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:51:25,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33855.95 MB 2025-02-14 20:51:25,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:51:25,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:51:25,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.96 seconds 2025-02-14 20:51:25,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:25,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19842.80 MB 2025-02-14 20:51:25,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33839.18 MB 2025-02-14 20:51:25,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13996.39 MB 2025-02-14 20:51:25,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42584.77 MB 2025-02-14 20:51:25,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38900.07 MB 2025-02-14 20:51:25,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3684.70 MB 2025-02-14 20:51:25,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33855.95 MB 2025-02-14 20:51:25,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:51:25,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:51:25,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:51:25,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:25,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33839.18 MB 2025-02-14 20:51:25,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24829.49 MB 2025-02-14 20:51:25,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9009.69 MB 2025-02-14 20:51:25,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38900.07 MB 2025-02-14 20:51:25,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38900.07 MB 2025-02-14 20:51:25,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:51:25,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36336.10 MB 2025-02-14 20:51:25,296 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 20:51:25,297 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:51:25,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:51:25,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:51:25,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:51:25,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:25,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24829.49 MB 2025-02-14 20:51:25,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33218.63 MB 2025-02-14 20:51:25,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 20:51:25,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38900.07 MB 2025-02-14 20:51:25,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43071.31 MB 2025-02-14 20:51:25,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 20:51:25,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33218.63 MB 2025-02-14 20:51:25,468 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 20:51:25,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:25,470 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:51:25,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:25,471 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:51:25,475 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:51:25,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:25,477 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:51:25,477 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:51:41,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:41,310 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:51:41,315 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:51:41,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:41,318 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 159, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:51:41,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:41,319 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 159, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:51:43,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:51:43,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:51:43,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.48 seconds 2025-02-14 20:51:43,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:43,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14076.64 MB 2025-02-14 20:51:43,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14639.34 MB 2025-02-14 20:51:43,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 562.69 MB 2025-02-14 20:51:43,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51413.78 MB 2025-02-14 20:51:43,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 20:51:43,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31883.00 MB 2025-02-14 20:51:43,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23548.02 MB 2025-02-14 20:51:43,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:51:43,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:51:43,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:51:43,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:43,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14639.34 MB 2025-02-14 20:51:43,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14911.96 MB 2025-02-14 20:51:43,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.62 MB 2025-02-14 20:51:43,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 20:51:43,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 20:51:43,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:51:43,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.81 MB 2025-02-14 20:51:44,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:51:44,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:51:44,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 20:51:44,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:44,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14911.96 MB 2025-02-14 20:51:44,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15123.06 MB 2025-02-14 20:51:44,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.10 MB 2025-02-14 20:51:44,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 20:51:44,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19058.92 MB 2025-02-14 20:51:44,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 20:51:44,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19082.65 MB 2025-02-14 20:51:44,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:51:44,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:51:44,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:51:44,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:44,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15122.99 MB 2025-02-14 20:51:44,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15873.90 MB 2025-02-14 20:51:44,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.91 MB 2025-02-14 20:51:44,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-14 20:51:44,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19058.92 MB 2025-02-14 20:51:44,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:51:44,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16437.33 MB 2025-02-14 20:51:44,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:51:44,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:51:44,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:51:44,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:44,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.90 MB 2025-02-14 20:51:44,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16765.08 MB 2025-02-14 20:51:44,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 891.18 MB 2025-02-14 20:51:44,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-14 20:51:44,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19998.44 MB 2025-02-14 20:51:44,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 939.52 MB 2025-02-14 20:51:44,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18969.41 MB 2025-02-14 20:51:44,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:51:44,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:51:44,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:51:44,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:44,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15122.99 MB 2025-02-14 20:51:44,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16765.08 MB 2025-02-14 20:51:44,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.08 MB 2025-02-14 20:51:44,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-14 20:51:44,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19998.44 MB 2025-02-14 20:51:44,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 939.52 MB 2025-02-14 20:51:44,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18969.41 MB 2025-02-14 20:51:44,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:51:44,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:51:44,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:51:44,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:44,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17374.66 MB 2025-02-14 20:51:44,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17679.54 MB 2025-02-14 20:51:44,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.88 MB 2025-02-14 20:51:44,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19998.44 MB 2025-02-14 20:51:44,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20162.02 MB 2025-02-14 20:51:44,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 20:51:44,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17968.61 MB 2025-02-14 20:51:44,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:51:44,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:51:44,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:51:44,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:44,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17843.67 MB 2025-02-14 20:51:44,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18064.42 MB 2025-02-14 20:51:44,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.75 MB 2025-02-14 20:51:44,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20162.02 MB 2025-02-14 20:51:44,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20162.02 MB 2025-02-14 20:51:44,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:51:44,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18081.47 MB 2025-02-14 20:51:44,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:51:44,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:51:44,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.44 seconds 2025-02-14 20:51:44,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:44,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13522.68 MB 2025-02-14 20:51:44,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18265.35 MB 2025-02-14 20:51:44,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.67 MB 2025-02-14 20:51:44,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51413.78 MB 2025-02-14 20:51:44,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20162.02 MB 2025-02-14 20:51:44,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31251.76 MB 2025-02-14 20:51:44,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18265.35 MB 2025-02-14 20:51:45,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:51:45,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:51:45,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:51:45,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:45,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18265.35 MB 2025-02-14 20:51:45,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17387.43 MB 2025-02-14 20:51:45,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -877.92 MB 2025-02-14 20:51:45,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20162.02 MB 2025-02-14 20:51:45,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20162.02 MB 2025-02-14 20:51:45,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:51:45,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19068.49 MB 2025-02-14 20:51:45,044 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 20:51:45,044 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1,'] 2025-02-14 20:51:45,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:51:45,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:51:45,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:51:45,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:51:45,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17387.43 MB 2025-02-14 20:51:45,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.73 MB 2025-02-14 20:51:45,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 20:51:45,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20162.02 MB 2025-02-14 20:51:45,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30643.59 MB 2025-02-14 20:51:45,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 20:51:45,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25820.73 MB 2025-02-14 20:51:45,212 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 20:51:45,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:45,214 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:51:45,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:45,215 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:51:45,219 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:51:45,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:51:45,220 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:51:45,220 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1,'] 2025-02-14 20:52:09,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:09,277 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:52:09,282 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:52:09,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:09,285 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:52:09,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:09,286 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:52:13,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:52:13,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:52:13,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.38 seconds 2025-02-14 20:52:13,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:13,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.70 MB 2025-02-14 20:52:13,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15942.22 MB 2025-02-14 20:52:13,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1001.52 MB 2025-02-14 20:52:13,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39028.00 MB 2025-02-14 20:52:13,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20210.25 MB 2025-02-14 20:52:13,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18817.74 MB 2025-02-14 20:52:13,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24865.05 MB 2025-02-14 20:52:13,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:52:13,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:52:13,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:52:13,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:13,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15942.22 MB 2025-02-14 20:52:13,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15942.80 MB 2025-02-14 20:52:13,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 0.58 MB 2025-02-14 20:52:13,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20210.25 MB 2025-02-14 20:52:13,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20969.42 MB 2025-02-14 20:52:13,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 759.17 MB 2025-02-14 20:52:13,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18987.00 MB 2025-02-14 20:52:14,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:52:14,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:52:14,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.02 seconds 2025-02-14 20:52:14,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:14,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15942.80 MB 2025-02-14 20:52:14,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16226.80 MB 2025-02-14 20:52:14,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 284.00 MB 2025-02-14 20:52:14,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20969.42 MB 2025-02-14 20:52:14,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20969.42 MB 2025-02-14 20:52:14,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:14,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20197.38 MB 2025-02-14 20:52:14,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:52:14,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:52:14,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:52:14,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:14,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16226.80 MB 2025-02-14 20:52:14,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17237.45 MB 2025-02-14 20:52:14,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1010.66 MB 2025-02-14 20:52:14,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20969.42 MB 2025-02-14 20:52:14,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20969.42 MB 2025-02-14 20:52:14,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:14,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17995.78 MB 2025-02-14 20:52:14,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:52:14,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:52:14,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 20:52:14,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:14,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17237.45 MB 2025-02-14 20:52:14,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18436.88 MB 2025-02-14 20:52:14,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1199.42 MB 2025-02-14 20:52:14,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20969.42 MB 2025-02-14 20:52:14,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22739.42 MB 2025-02-14 20:52:14,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1770.00 MB 2025-02-14 20:52:14,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21404.61 MB 2025-02-14 20:52:14,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:52:14,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:52:14,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 20:52:14,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:14,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16226.80 MB 2025-02-14 20:52:14,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18436.88 MB 2025-02-14 20:52:14,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2210.08 MB 2025-02-14 20:52:14,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20969.42 MB 2025-02-14 20:52:14,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22739.42 MB 2025-02-14 20:52:14,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1770.00 MB 2025-02-14 20:52:14,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21404.61 MB 2025-02-14 20:52:14,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:52:14,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:52:14,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:52:14,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:14,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19257.32 MB 2025-02-14 20:52:14,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19667.67 MB 2025-02-14 20:52:14,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.35 MB 2025-02-14 20:52:14,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22739.42 MB 2025-02-14 20:52:14,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22961.72 MB 2025-02-14 20:52:14,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 222.30 MB 2025-02-14 20:52:14,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20047.27 MB 2025-02-14 20:52:14,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:52:14,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:52:14,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:52:14,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:14,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19888.57 MB 2025-02-14 20:52:14,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20108.50 MB 2025-02-14 20:52:14,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.94 MB 2025-02-14 20:52:14,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22961.72 MB 2025-02-14 20:52:14,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22961.72 MB 2025-02-14 20:52:14,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:14,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20142.47 MB 2025-02-14 20:52:14,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:52:14,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:52:14,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.65 seconds 2025-02-14 20:52:14,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:14,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13954.70 MB 2025-02-14 20:52:14,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20309.01 MB 2025-02-14 20:52:14,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6354.31 MB 2025-02-14 20:52:14,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39028.00 MB 2025-02-14 20:52:14,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22961.72 MB 2025-02-14 20:52:14,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16066.28 MB 2025-02-14 20:52:14,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20309.01 MB 2025-02-14 20:52:15,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:52:15,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:52:15,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 20:52:15,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:15,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15066.72 MB 2025-02-14 20:52:15,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18072.28 MB 2025-02-14 20:52:15,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-14 20:52:15,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22961.72 MB 2025-02-14 20:52:15,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22961.72 MB 2025-02-14 20:52:15,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:15,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18372.80 MB 2025-02-14 20:52:15,220 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 20:52:15,220 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:52:15,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:52:15,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:52:15,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:52:15,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:15,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18072.28 MB 2025-02-14 20:52:15,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26487.23 MB 2025-02-14 20:52:15,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 20:52:15,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22961.72 MB 2025-02-14 20:52:15,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31329.35 MB 2025-02-14 20:52:15,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 20:52:15,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26487.23 MB 2025-02-14 20:52:15,382 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 20:52:15,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:15,384 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:52:15,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:15,385 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:52:15,389 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:52:15,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:15,390 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:52:15,390 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:52:23,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:23,742 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:52:23,746 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:52:23,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:23,750 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 462, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:52:23,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:23,751 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 462, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:52:30,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:52:30,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:52:30,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.15 seconds 2025-02-14 20:52:30,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:30,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16188.00 MB 2025-02-14 20:52:30,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17822.99 MB 2025-02-14 20:52:30,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1634.99 MB 2025-02-14 20:52:30,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39696.99 MB 2025-02-14 20:52:30,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20208.16 MB 2025-02-14 20:52:30,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19488.83 MB 2025-02-14 20:52:30,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26791.83 MB 2025-02-14 20:52:30,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:52:30,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:52:30,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 20:52:30,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:30,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17822.99 MB 2025-02-14 20:52:30,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18180.69 MB 2025-02-14 20:52:30,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 357.70 MB 2025-02-14 20:52:30,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20208.16 MB 2025-02-14 20:52:30,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28682.75 MB 2025-02-14 20:52:30,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8474.59 MB 2025-02-14 20:52:30,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25252.69 MB 2025-02-14 20:52:32,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:52:32,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:52:32,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 20:52:32,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:32,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18180.69 MB 2025-02-14 20:52:32,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18711.53 MB 2025-02-14 20:52:32,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:52:32,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28682.75 MB 2025-02-14 20:52:32,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22332.57 MB 2025-02-14 20:52:32,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6350.18 MB 2025-02-14 20:52:32,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22691.12 MB 2025-02-14 20:52:32,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:52:32,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:52:32,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:52:32,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:32,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18711.53 MB 2025-02-14 20:52:32,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20601.07 MB 2025-02-14 20:52:32,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:52:32,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22332.57 MB 2025-02-14 20:52:32,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24220.01 MB 2025-02-14 20:52:32,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 20:52:32,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22018.50 MB 2025-02-14 20:52:33,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:52:33,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:52:33,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:52:33,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:33,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20601.07 MB 2025-02-14 20:52:33,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22842.92 MB 2025-02-14 20:52:33,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:52:33,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24220.01 MB 2025-02-14 20:52:33,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 20:52:33,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 20:52:33,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28387.21 MB 2025-02-14 20:52:33,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:52:33,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:52:33,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:52:33,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:33,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18711.53 MB 2025-02-14 20:52:33,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22842.92 MB 2025-02-14 20:52:33,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:52:33,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22332.57 MB 2025-02-14 20:52:33,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 20:52:33,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-14 20:52:33,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28387.21 MB 2025-02-14 20:52:33,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:52:33,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:52:33,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:52:33,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:33,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24376.47 MB 2025-02-14 20:52:33,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25143.47 MB 2025-02-14 20:52:33,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:52:33,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30356.28 MB 2025-02-14 20:52:33,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 20:52:33,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:52:33,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25851.26 MB 2025-02-14 20:52:33,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:52:33,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:52:33,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:52:33,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:33,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25556.36 MB 2025-02-14 20:52:33,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25784.03 MB 2025-02-14 20:52:33,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.67 MB 2025-02-14 20:52:33,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30773.61 MB 2025-02-14 20:52:33,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 20:52:33,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:33,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25988.21 MB 2025-02-14 20:52:33,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:52:33,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:52:33,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.53 seconds 2025-02-14 20:52:33,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:33,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-14 20:52:33,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25985.10 MB 2025-02-14 20:52:33,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11406.75 MB 2025-02-14 20:52:33,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39696.99 MB 2025-02-14 20:52:33,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 20:52:33,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8923.38 MB 2025-02-14 20:52:33,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25988.21 MB 2025-02-14 20:52:33,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:52:33,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:52:33,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:52:33,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:33,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25985.10 MB 2025-02-14 20:52:33,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19582.74 MB 2025-02-14 20:52:33,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6402.36 MB 2025-02-14 20:52:33,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30773.61 MB 2025-02-14 20:52:33,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 20:52:33,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:33,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28496.77 MB 2025-02-14 20:52:33,566 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 20:52:33,566 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:52:33,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:52:33,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:52:33,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:52:33,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:33,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19582.74 MB 2025-02-14 20:52:33,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28021.76 MB 2025-02-14 20:52:33,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 20:52:33,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30773.61 MB 2025-02-14 20:52:33,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41263.56 MB 2025-02-14 20:52:33,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 20:52:33,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28021.76 MB 2025-02-14 20:52:33,732 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 20:52:33,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:33,734 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:52:33,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:33,735 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:52:33,740 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:52:33,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:33,741 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:52:33,741 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:52:42,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:42,916 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:52:42,920 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:52:42,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:42,924 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 144, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:52:42,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:42,925 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 144, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:52:45,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:52:45,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:52:45,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.29 seconds 2025-02-14 20:52:45,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:45,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13972.12 MB 2025-02-14 20:52:45,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14481.73 MB 2025-02-14 20:52:45,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 509.61 MB 2025-02-14 20:52:45,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53848.57 MB 2025-02-14 20:52:45,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19293.80 MB 2025-02-14 20:52:45,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34554.77 MB 2025-02-14 20:52:45,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23443.49 MB 2025-02-14 20:52:45,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:52:45,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:52:45,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:52:45,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:45,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14481.73 MB 2025-02-14 20:52:45,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14728.63 MB 2025-02-14 20:52:45,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.90 MB 2025-02-14 20:52:45,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19293.80 MB 2025-02-14 20:52:45,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19293.80 MB 2025-02-14 20:52:45,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:45,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16561.05 MB 2025-02-14 20:52:45,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:52:45,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:52:45,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.69 seconds 2025-02-14 20:52:45,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:45,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14728.63 MB 2025-02-14 20:52:45,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14919.74 MB 2025-02-14 20:52:45,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.10 MB 2025-02-14 20:52:45,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19293.80 MB 2025-02-14 20:52:45,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19293.80 MB 2025-02-14 20:52:45,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:45,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18898.28 MB 2025-02-14 20:52:45,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:52:45,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:52:45,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:52:45,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:45,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14919.67 MB 2025-02-14 20:52:45,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15599.74 MB 2025-02-14 20:52:45,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 680.07 MB 2025-02-14 20:52:45,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19293.80 MB 2025-02-14 20:52:45,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19293.80 MB 2025-02-14 20:52:45,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:45,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16110.02 MB 2025-02-14 20:52:46,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:52:46,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:52:46,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:52:46,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:46,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15599.74 MB 2025-02-14 20:52:46,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16406.85 MB 2025-02-14 20:52:46,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 807.11 MB 2025-02-14 20:52:46,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19293.80 MB 2025-02-14 20:52:46,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19633.54 MB 2025-02-14 20:52:46,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 339.74 MB 2025-02-14 20:52:46,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18402.75 MB 2025-02-14 20:52:46,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:52:46,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:52:46,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:52:46,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:46,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14919.67 MB 2025-02-14 20:52:46,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16406.85 MB 2025-02-14 20:52:46,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1487.18 MB 2025-02-14 20:52:46,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19293.80 MB 2025-02-14 20:52:46,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19633.54 MB 2025-02-14 20:52:46,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 339.74 MB 2025-02-14 20:52:46,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18402.75 MB 2025-02-14 20:52:46,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:52:46,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:52:46,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 20:52:46,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:46,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16958.92 MB 2025-02-14 20:52:46,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17235.04 MB 2025-02-14 20:52:46,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.12 MB 2025-02-14 20:52:46,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19633.54 MB 2025-02-14 20:52:46,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19782.43 MB 2025-02-14 20:52:46,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 20:52:46,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17499.58 MB 2025-02-14 20:52:46,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:52:46,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:52:46,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:52:46,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:46,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.69 MB 2025-02-14 20:52:46,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17611.70 MB 2025-02-14 20:52:46,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.01 MB 2025-02-14 20:52:46,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19782.43 MB 2025-02-14 20:52:46,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19782.43 MB 2025-02-14 20:52:46,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:46,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17612.65 MB 2025-02-14 20:52:46,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:52:46,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:52:46,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.16 seconds 2025-02-14 20:52:46,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:46,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13470.41 MB 2025-02-14 20:52:46,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17812.72 MB 2025-02-14 20:52:46,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4342.31 MB 2025-02-14 20:52:46,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53848.57 MB 2025-02-14 20:52:46,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19782.43 MB 2025-02-14 20:52:46,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34066.14 MB 2025-02-14 20:52:46,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17812.72 MB 2025-02-14 20:52:46,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:52:46,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:52:46,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:52:46,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:46,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17812.72 MB 2025-02-14 20:52:46,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17265.90 MB 2025-02-14 20:52:46,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -546.82 MB 2025-02-14 20:52:46,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19782.43 MB 2025-02-14 20:52:46,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19782.43 MB 2025-02-14 20:52:46,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:52:46,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19018.03 MB 2025-02-14 20:52:46,367 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 20:52:46,367 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:52:46,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:52:46,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:52:46,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:52:46,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:52:46,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17265.90 MB 2025-02-14 20:52:46,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25703.37 MB 2025-02-14 20:52:46,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 20:52:46,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19782.43 MB 2025-02-14 20:52:46,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30268.19 MB 2025-02-14 20:52:46,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 20:52:46,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25703.37 MB 2025-02-14 20:52:46,535 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 20:52:46,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:46,536 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:52:46,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:46,537 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:52:46,542 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:52:46,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:52:46,543 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:52:46,543 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:53:43,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:53:43,807 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:53:43,815 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:53:43,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:53:43,821 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:53:43,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:53:43,823 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:53:46,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:53:46,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:53:46,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-14 20:53:46,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:46,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14041.80 MB 2025-02-14 20:53:46,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14586.80 MB 2025-02-14 20:53:46,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-14 20:53:46,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38656.80 MB 2025-02-14 20:53:46,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21730.69 MB 2025-02-14 20:53:46,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16926.11 MB 2025-02-14 20:53:46,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23513.17 MB 2025-02-14 20:53:46,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:53:46,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:53:46,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:53:46,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:46,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14586.80 MB 2025-02-14 20:53:46,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14710.39 MB 2025-02-14 20:53:46,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.59 MB 2025-02-14 20:53:46,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21730.69 MB 2025-02-14 20:53:46,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21730.69 MB 2025-02-14 20:53:46,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:53:46,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16504.43 MB 2025-02-14 20:53:46,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:53:46,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:53:46,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.67 seconds 2025-02-14 20:53:46,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:46,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14710.39 MB 2025-02-14 20:53:46,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14888.22 MB 2025-02-14 20:53:46,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-14 20:53:46,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21730.69 MB 2025-02-14 20:53:46,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 20:53:46,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -339.74 MB 2025-02-14 20:53:46,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18880.04 MB 2025-02-14 20:53:46,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:53:46,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:53:46,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 20:53:46,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:46,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14888.16 MB 2025-02-14 20:53:46,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15521.00 MB 2025-02-14 20:53:46,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-14 20:53:46,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 20:53:46,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 20:53:46,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:53:46,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15995.84 MB 2025-02-14 20:53:47,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:53:47,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:53:47,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:53:47,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:47,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15521.00 MB 2025-02-14 20:53:47,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16272.06 MB 2025-02-14 20:53:47,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-14 20:53:47,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 20:53:47,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 20:53:47,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:53:47,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18129.35 MB 2025-02-14 20:53:47,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:53:47,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:53:47,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 20:53:47,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:47,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14888.16 MB 2025-02-14 20:53:47,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16272.06 MB 2025-02-14 20:53:47,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-14 20:53:47,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 20:53:47,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 20:53:47,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:53:47,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18129.35 MB 2025-02-14 20:53:47,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:53:47,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:53:47,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 20:53:47,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:47,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16785.80 MB 2025-02-14 20:53:47,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17042.74 MB 2025-02-14 20:53:47,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-14 20:53:47,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 20:53:47,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21529.36 MB 2025-02-14 20:53:47,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 138.41 MB 2025-02-14 20:53:47,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17290.97 MB 2025-02-14 20:53:47,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:53:47,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:53:47,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:53:47,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:47,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17181.07 MB 2025-02-14 20:53:47,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17401.27 MB 2025-02-14 20:53:47,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.20 MB 2025-02-14 20:53:47,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21529.36 MB 2025-02-14 20:53:47,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21529.36 MB 2025-02-14 20:53:47,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:53:47,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17401.27 MB 2025-02-14 20:53:47,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:53:47,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:53:47,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.33 seconds 2025-02-14 20:53:47,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:47,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13505.25 MB 2025-02-14 20:53:47,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14240.09 MB 2025-02-14 20:53:47,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.84 MB 2025-02-14 20:53:47,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38656.80 MB 2025-02-14 20:53:47,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21529.36 MB 2025-02-14 20:53:47,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17127.44 MB 2025-02-14 20:53:47,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17601.97 MB 2025-02-14 20:53:47,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:53:47,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:53:47,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 20:53:47,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:47,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14240.09 MB 2025-02-14 20:53:47,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17248.60 MB 2025-02-14 20:53:47,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-14 20:53:47,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21529.36 MB 2025-02-14 20:53:47,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21529.36 MB 2025-02-14 20:53:47,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:53:47,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17549.41 MB 2025-02-14 20:53:47,457 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 20:53:47,458 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:53:47,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:53:47,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:53:47,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:53:47,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:53:47,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17248.60 MB 2025-02-14 20:53:47,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.80 MB 2025-02-14 20:53:47,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 20:53:47,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21529.36 MB 2025-02-14 20:53:47,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29905.39 MB 2025-02-14 20:53:47,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 20:53:47,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25671.80 MB 2025-02-14 20:53:47,620 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 20:53:47,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:53:47,622 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:53:47,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:53:47,622 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:53:47,627 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:53:47,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:53:47,629 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:53:47,629 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:54:40,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:54:40,771 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:54:40,776 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:54:40,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:54:40,780 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1099, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:54:40,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:54:40,781 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1099, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:54:57,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:54:57,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:54:57,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.81 seconds 2025-02-14 20:54:57,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:54:57,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20626.71 MB 2025-02-14 20:54:57,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24516.93 MB 2025-02-14 20:54:57,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3890.22 MB 2025-02-14 20:54:57,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38281.41 MB 2025-02-14 20:54:57,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28288.48 MB 2025-02-14 20:54:57,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9992.93 MB 2025-02-14 20:54:57,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.28 MB 2025-02-14 20:54:57,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:54:57,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:54:57,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 20:54:57,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:54:57,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24516.93 MB 2025-02-14 20:54:57,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21492.26 MB 2025-02-14 20:54:57,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3024.67 MB 2025-02-14 20:54:57,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28288.48 MB 2025-02-14 20:54:57,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43885.00 MB 2025-02-14 20:54:57,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15596.52 MB 2025-02-14 20:54:57,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36501.90 MB 2025-02-14 20:54:59,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:54:59,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:54:59,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 20:54:59,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:54:59,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21492.26 MB 2025-02-14 20:54:59,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22023.10 MB 2025-02-14 20:54:59,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:54:59,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43885.00 MB 2025-02-14 20:54:59,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24425.53 MB 2025-02-14 20:54:59,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19459.47 MB 2025-02-14 20:54:59,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26002.68 MB 2025-02-14 20:54:59,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:54:59,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:54:59,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:54:59,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:54:59,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22023.10 MB 2025-02-14 20:54:59,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23912.63 MB 2025-02-14 20:54:59,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:54:59,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24425.53 MB 2025-02-14 20:54:59,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27256.68 MB 2025-02-14 20:54:59,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 20:54:59,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25330.06 MB 2025-02-14 20:54:59,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:54:59,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:54:59,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 20:54:59,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:54:59,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23912.63 MB 2025-02-14 20:54:59,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26154.49 MB 2025-02-14 20:54:59,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:54:59,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27256.68 MB 2025-02-14 20:54:59,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33390.85 MB 2025-02-14 20:54:59,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:54:59,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31698.77 MB 2025-02-14 20:54:59,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:54:59,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:54:59,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:54:59,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:54:59,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22023.10 MB 2025-02-14 20:54:59,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26154.49 MB 2025-02-14 20:54:59,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:54:59,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24425.53 MB 2025-02-14 20:54:59,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33390.85 MB 2025-02-14 20:54:59,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 20:54:59,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31698.77 MB 2025-02-14 20:54:59,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:54:59,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:54:59,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:54:59,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:54:59,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27688.03 MB 2025-02-14 20:54:59,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28455.03 MB 2025-02-14 20:54:59,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:54:59,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33390.85 MB 2025-02-14 20:54:59,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33808.19 MB 2025-02-14 20:54:59,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:54:59,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29162.82 MB 2025-02-14 20:55:00,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:55:00,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:55:00,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:55:00,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:55:00,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28867.92 MB 2025-02-14 20:55:00,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29096.05 MB 2025-02-14 20:55:00,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-14 20:55:00,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33808.19 MB 2025-02-14 20:55:00,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33808.19 MB 2025-02-14 20:55:00,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:55:00,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29335.05 MB 2025-02-14 20:55:00,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:55:00,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:55:00,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.22 seconds 2025-02-14 20:55:00,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:55:00,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16797.71 MB 2025-02-14 20:55:00,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29296.53 MB 2025-02-14 20:55:00,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12498.82 MB 2025-02-14 20:55:00,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38281.41 MB 2025-02-14 20:55:00,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33808.19 MB 2025-02-14 20:55:00,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4473.23 MB 2025-02-14 20:55:00,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29335.05 MB 2025-02-14 20:55:00,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:55:00,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:55:00,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:55:00,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:55:00,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29296.53 MB 2025-02-14 20:55:00,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21786.54 MB 2025-02-14 20:55:00,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7509.99 MB 2025-02-14 20:55:00,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33808.19 MB 2025-02-14 20:55:00,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33808.19 MB 2025-02-14 20:55:00,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:55:00,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31795.30 MB 2025-02-14 20:55:00,289 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 20:55:00,290 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:55:00,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:55:00,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:55:00,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:55:00,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:55:00,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21786.54 MB 2025-02-14 20:55:00,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30183.18 MB 2025-02-14 20:55:00,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-14 20:55:00,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33808.19 MB 2025-02-14 20:55:00,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42154.85 MB 2025-02-14 20:55:00,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 20:55:00,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30183.18 MB 2025-02-14 20:55:00,454 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 20:55:00,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:55:00,455 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:55:00,456 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:55:00,456 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:55:00,461 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:55:00,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:55:00,462 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:55:00,462 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:56:37,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:56:37,760 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:56:37,765 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:56:37,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:56:37,768 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:56:37,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:56:37,769 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:56:55,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:56:55,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:56:55,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.15 seconds 2025-02-14 20:56:55,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:55,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21295.66 MB 2025-02-14 20:56:55,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25525.61 MB 2025-02-14 20:56:55,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4229.96 MB 2025-02-14 20:56:55,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50501.52 MB 2025-02-14 20:56:55,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30704.40 MB 2025-02-14 20:56:55,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19797.11 MB 2025-02-14 20:56:55,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34390.91 MB 2025-02-14 20:56:56,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:56:56,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:56:56,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 20:56:56,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:56,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25525.61 MB 2025-02-14 20:56:56,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21991.33 MB 2025-02-14 20:56:56,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3534.28 MB 2025-02-14 20:56:56,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30704.40 MB 2025-02-14 20:56:56,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44782.58 MB 2025-02-14 20:56:56,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14078.18 MB 2025-02-14 20:56:56,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38187.97 MB 2025-02-14 20:56:57,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:56:57,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:56:57,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 20:56:57,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:57,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21991.33 MB 2025-02-14 20:56:57,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22522.17 MB 2025-02-14 20:56:57,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:56:57,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44782.58 MB 2025-02-14 20:56:57,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28598.86 MB 2025-02-14 20:56:57,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16183.72 MB 2025-02-14 20:56:57,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26500.72 MB 2025-02-14 20:56:57,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:56:57,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:56:57,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:56:57,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:57,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22522.17 MB 2025-02-14 20:56:57,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24411.71 MB 2025-02-14 20:56:57,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:56:57,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 20:56:57,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28598.86 MB 2025-02-14 20:56:57,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:56:57,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25829.14 MB 2025-02-14 20:56:58,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:56:58,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:56:58,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:56:58,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:58,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24411.71 MB 2025-02-14 20:56:58,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26654.57 MB 2025-02-14 20:56:58,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.87 MB 2025-02-14 20:56:58,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 20:56:58,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 20:56:58,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 20:56:58,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32198.85 MB 2025-02-14 20:56:58,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:56:58,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:56:58,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:56:58,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:58,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22522.17 MB 2025-02-14 20:56:58,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26654.57 MB 2025-02-14 20:56:58,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.40 MB 2025-02-14 20:56:58,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 20:56:58,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 20:56:58,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 20:56:58,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32198.85 MB 2025-02-14 20:56:58,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:56:58,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:56:58,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:56:58,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:58,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28188.11 MB 2025-02-14 20:56:58,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28955.12 MB 2025-02-14 20:56:58,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:56:58,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34261.17 MB 2025-02-14 20:56:58,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 20:56:58,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 20:56:58,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29662.91 MB 2025-02-14 20:56:58,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:56:58,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:56:58,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:56:58,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:58,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29368.01 MB 2025-02-14 20:56:58,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29596.92 MB 2025-02-14 20:56:58,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 20:56:58,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 20:56:58,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 20:56:58,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:56:58,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29822.64 MB 2025-02-14 20:56:58,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:56:58,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:56:58,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.54 seconds 2025-02-14 20:56:58,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:58,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17132.18 MB 2025-02-14 20:56:58,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29796.91 MB 2025-02-14 20:56:58,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12664.73 MB 2025-02-14 20:56:58,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50501.52 MB 2025-02-14 20:56:58,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 20:56:58,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15823.01 MB 2025-02-14 20:56:58,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29822.64 MB 2025-02-14 20:56:58,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:56:58,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:56:58,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 20:56:58,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:58,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29796.91 MB 2025-02-14 20:56:58,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22120.30 MB 2025-02-14 20:56:58,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7676.61 MB 2025-02-14 20:56:58,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 20:56:58,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 20:56:58,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:56:58,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32295.55 MB 2025-02-14 20:56:58,599 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 20:56:58,600 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 20:56:58,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:56:58,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:56:58,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:56:58,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:56:58,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22120.30 MB 2025-02-14 20:56:58,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30513.58 MB 2025-02-14 20:56:58,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 20:56:58,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 20:56:58,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-14 20:56:58,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 20:56:58,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30513.58 MB 2025-02-14 20:56:58,765 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 20:56:58,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:56:58,767 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:56:58,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:56:58,768 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:56:58,772 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:56:58,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:56:58,773 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:56:58,773 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 20:58:35,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:58:35,270 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 20:58:35,278 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 20:58:35,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:58:35,285 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2002, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 20:58:35,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:58:35,287 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2002, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 20:59:06,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 20:59:06,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 20:59:06,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.78 seconds 2025-02-14 20:59:06,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:06,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26918.96 MB 2025-02-14 20:59:06,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34003.93 MB 2025-02-14 20:59:06,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7084.97 MB 2025-02-14 20:59:06,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51371.84 MB 2025-02-14 20:59:06,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37721.47 MB 2025-02-14 20:59:06,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13650.36 MB 2025-02-14 20:59:06,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42960.73 MB 2025-02-14 20:59:06,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 20:59:06,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 20:59:06,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 20:59:06,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:06,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34003.93 MB 2025-02-14 20:59:06,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26186.67 MB 2025-02-14 20:59:06,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7817.26 MB 2025-02-14 20:59:06,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37721.47 MB 2025-02-14 20:59:06,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64804.09 MB 2025-02-14 20:59:06,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 27082.62 MB 2025-02-14 20:59:06,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54167.69 MB 2025-02-14 20:59:08,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 20:59:08,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 20:59:08,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 20:59:08,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26186.67 MB 2025-02-14 20:59:08,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26717.51 MB 2025-02-14 20:59:08,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 20:59:08,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64804.09 MB 2025-02-14 20:59:08,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28586.28 MB 2025-02-14 20:59:08,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36217.82 MB 2025-02-14 20:59:08,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30697.10 MB 2025-02-14 20:59:08,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 20:59:08,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 20:59:08,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 20:59:08,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26717.51 MB 2025-02-14 20:59:08,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28607.05 MB 2025-02-14 20:59:08,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 20:59:08,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28586.28 MB 2025-02-14 20:59:08,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31889.29 MB 2025-02-14 20:59:08,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 20:59:08,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.48 MB 2025-02-14 20:59:08,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 20:59:08,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 20:59:08,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 20:59:08,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28607.05 MB 2025-02-14 20:59:08,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30848.90 MB 2025-02-14 20:59:08,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 20:59:08,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31889.29 MB 2025-02-14 20:59:08,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38023.46 MB 2025-02-14 20:59:08,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 20:59:08,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36393.19 MB 2025-02-14 20:59:08,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 20:59:08,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 20:59:08,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 20:59:08,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26717.51 MB 2025-02-14 20:59:08,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30848.90 MB 2025-02-14 20:59:08,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 20:59:08,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28586.28 MB 2025-02-14 20:59:08,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38023.46 MB 2025-02-14 20:59:08,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 20:59:08,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36393.19 MB 2025-02-14 20:59:08,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 20:59:08,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 20:59:08,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 20:59:08,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32382.45 MB 2025-02-14 20:59:08,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33149.45 MB 2025-02-14 20:59:08,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 20:59:08,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38023.46 MB 2025-02-14 20:59:08,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38438.70 MB 2025-02-14 20:59:08,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 20:59:08,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33857.24 MB 2025-02-14 20:59:08,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 20:59:08,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 20:59:08,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:59:08,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33562.34 MB 2025-02-14 20:59:08,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33791.34 MB 2025-02-14 20:59:08,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.00 MB 2025-02-14 20:59:08,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38438.70 MB 2025-02-14 20:59:08,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38438.70 MB 2025-02-14 20:59:08,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:59:08,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33992.47 MB 2025-02-14 20:59:08,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 20:59:08,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 20:59:08,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.28 seconds 2025-02-14 20:59:08,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19943.83 MB 2025-02-14 20:59:08,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33991.57 MB 2025-02-14 20:59:08,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14047.74 MB 2025-02-14 20:59:08,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51371.84 MB 2025-02-14 20:59:08,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38438.70 MB 2025-02-14 20:59:08,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12933.14 MB 2025-02-14 20:59:08,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33992.47 MB 2025-02-14 20:59:08,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 20:59:08,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 20:59:08,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 20:59:08,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33991.57 MB 2025-02-14 20:59:08,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24935.52 MB 2025-02-14 20:59:08,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9056.05 MB 2025-02-14 20:59:08,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38438.70 MB 2025-02-14 20:59:08,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38438.70 MB 2025-02-14 20:59:08,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 20:59:08,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36492.79 MB 2025-02-14 20:59:08,853 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 20:59:08,854 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 20:59:08,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 20:59:08,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 20:59:08,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 20:59:08,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 20:59:08,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24935.52 MB 2025-02-14 20:59:08,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33340.60 MB 2025-02-14 20:59:08,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-14 20:59:08,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38438.70 MB 2025-02-14 20:59:08,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46793.75 MB 2025-02-14 20:59:08,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 20:59:08,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33340.60 MB 2025-02-14 20:59:09,016 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 20:59:09,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:59:09,018 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 20:59:09,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:59:09,019 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 20:59:09,023 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 20:59:09,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 20:59:09,025 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 20:59:09,025 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:00:04,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:00:04,443 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:00:04,448 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:00:04,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:00:04,452 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1941, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:00:04,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:00:04,453 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1941, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:00:34,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:00:34,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:00:34,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.98 seconds 2025-02-14 21:00:34,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:34,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26493.90 MB 2025-02-14 21:00:34,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33363.00 MB 2025-02-14 21:00:34,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6869.09 MB 2025-02-14 21:00:34,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55148.81 MB 2025-02-14 21:00:34,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37507.56 MB 2025-02-14 21:00:34,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17641.24 MB 2025-02-14 21:00:34,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42307.87 MB 2025-02-14 21:00:34,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:00:34,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:00:34,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 21:00:34,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:34,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33363.00 MB 2025-02-14 21:00:34,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25868.50 MB 2025-02-14 21:00:34,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7494.49 MB 2025-02-14 21:00:34,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37507.56 MB 2025-02-14 21:00:34,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62142.81 MB 2025-02-14 21:00:34,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24635.24 MB 2025-02-14 21:00:34,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52548.67 MB 2025-02-14 21:00:36,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:00:36,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:00:36,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:00:36,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:36,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25868.50 MB 2025-02-14 21:00:36,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26399.35 MB 2025-02-14 21:00:36,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:00:36,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62142.81 MB 2025-02-14 21:00:36,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32052.87 MB 2025-02-14 21:00:36,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30089.94 MB 2025-02-14 21:00:36,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30378.02 MB 2025-02-14 21:00:36,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:00:36,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:00:36,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:00:36,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:36,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-14 21:00:36,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28288.88 MB 2025-02-14 21:00:36,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:00:36,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32052.87 MB 2025-02-14 21:00:36,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32052.87 MB 2025-02-14 21:00:36,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:00:36,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29706.31 MB 2025-02-14 21:00:36,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:00:36,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:00:36,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:00:36,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:36,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28288.88 MB 2025-02-14 21:00:36,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-14 21:00:36,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:00:36,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32052.87 MB 2025-02-14 21:00:36,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37715.18 MB 2025-02-14 21:00:36,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:00:36,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-14 21:00:36,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:00:36,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:00:36,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:00:36,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:36,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-14 21:00:36,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-14 21:00:36,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:00:36,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32052.87 MB 2025-02-14 21:00:36,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37715.18 MB 2025-02-14 21:00:36,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:00:36,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-14 21:00:37,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:00:37,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:00:37,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.34 seconds 2025-02-14 21:00:37,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:37,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32064.28 MB 2025-02-14 21:00:37,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32831.28 MB 2025-02-14 21:00:37,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:00:37,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37715.18 MB 2025-02-14 21:00:37,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38132.51 MB 2025-02-14 21:00:37,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:00:37,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33539.07 MB 2025-02-14 21:00:37,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:00:37,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:00:37,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:00:37,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:37,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33244.17 MB 2025-02-14 21:00:37,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33474.24 MB 2025-02-14 21:00:37,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.07 MB 2025-02-14 21:00:37,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38132.51 MB 2025-02-14 21:00:37,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38132.51 MB 2025-02-14 21:00:37,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:00:37,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33678.46 MB 2025-02-14 21:00:37,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:00:37,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:00:37,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.64 seconds 2025-02-14 21:00:37,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:37,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19731.31 MB 2025-02-14 21:00:37,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33675.31 MB 2025-02-14 21:00:37,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13944.00 MB 2025-02-14 21:00:37,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55148.81 MB 2025-02-14 21:00:37,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38132.51 MB 2025-02-14 21:00:37,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17016.29 MB 2025-02-14 21:00:37,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33678.46 MB 2025-02-14 21:00:37,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:00:37,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:00:37,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:00:37,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:37,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33675.31 MB 2025-02-14 21:00:37,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36689.34 MB 2025-02-14 21:00:37,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 21:00:37,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38132.51 MB 2025-02-14 21:00:37,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38132.51 MB 2025-02-14 21:00:37,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:00:37,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36990.71 MB 2025-02-14 21:00:37,385 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:00:37,385 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:00:37,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:00:37,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:00:37,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:00:37,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:00:37,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36689.34 MB 2025-02-14 21:00:37,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45128.37 MB 2025-02-14 21:00:37,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:00:37,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38132.51 MB 2025-02-14 21:00:37,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48622.47 MB 2025-02-14 21:00:37,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 21:00:37,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45128.37 MB 2025-02-14 21:00:37,547 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:00:37,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:00:37,549 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:00:37,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:00:37,550 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:00:37,554 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:00:37,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:00:37,555 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:00:37,555 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:01:55,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:01:55,468 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:01:55,473 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:01:55,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:01:55,477 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1322, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:01:55,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:01:55,478 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1322, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:02:15,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:02:15,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:02:15,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.32 seconds 2025-02-14 21:02:15,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:15,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22180.61 MB 2025-02-14 21:02:15,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26859.36 MB 2025-02-14 21:02:15,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4678.75 MB 2025-02-14 21:02:15,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61209.58 MB 2025-02-14 21:02:15,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35355.89 MB 2025-02-14 21:02:15,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25853.69 MB 2025-02-14 21:02:15,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35728.85 MB 2025-02-14 21:02:15,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:02:15,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:02:15,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:02:15,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:15,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26859.36 MB 2025-02-14 21:02:15,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22650.52 MB 2025-02-14 21:02:15,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4208.84 MB 2025-02-14 21:02:15,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35355.89 MB 2025-02-14 21:02:15,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45451.58 MB 2025-02-14 21:02:15,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10095.69 MB 2025-02-14 21:02:15,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40720.80 MB 2025-02-14 21:02:17,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:02:17,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:02:17,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 21:02:17,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:17,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22650.52 MB 2025-02-14 21:02:17,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23181.36 MB 2025-02-14 21:02:17,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:02:17,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45451.58 MB 2025-02-14 21:02:17,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-14 21:02:17,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14774.44 MB 2025-02-14 21:02:17,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27159.91 MB 2025-02-14 21:02:17,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:02:17,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:02:17,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:02:17,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:17,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23181.36 MB 2025-02-14 21:02:17,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25070.89 MB 2025-02-14 21:02:17,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:02:17,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 21:02:17,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-14 21:02:17,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:02:17,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26488.32 MB 2025-02-14 21:02:18,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:02:18,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:02:18,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:02:18,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:18,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25070.89 MB 2025-02-14 21:02:18,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27312.75 MB 2025-02-14 21:02:18,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:02:18,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 21:02:18,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35395.73 MB 2025-02-14 21:02:18,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:02:18,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32857.03 MB 2025-02-14 21:02:18,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:02:18,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:02:18,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:02:18,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:18,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23181.36 MB 2025-02-14 21:02:18,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27312.75 MB 2025-02-14 21:02:18,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:02:18,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 21:02:18,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35395.73 MB 2025-02-14 21:02:18,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:02:18,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32857.03 MB 2025-02-14 21:02:18,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:02:18,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:02:18,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:02:18,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:18,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28846.29 MB 2025-02-14 21:02:18,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29613.29 MB 2025-02-14 21:02:18,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:02:18,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35395.73 MB 2025-02-14 21:02:18,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35810.97 MB 2025-02-14 21:02:18,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:02:18,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30321.08 MB 2025-02-14 21:02:18,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:02:18,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:02:18,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:02:18,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:18,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30026.18 MB 2025-02-14 21:02:18,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30255.14 MB 2025-02-14 21:02:18,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 21:02:18,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35810.97 MB 2025-02-14 21:02:18,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35810.97 MB 2025-02-14 21:02:18,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:02:18,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.89 MB 2025-02-14 21:02:18,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:02:18,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:02:18,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.72 seconds 2025-02-14 21:02:18,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:18,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17574.66 MB 2025-02-14 21:02:18,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30455.72 MB 2025-02-14 21:02:18,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12881.06 MB 2025-02-14 21:02:18,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61209.58 MB 2025-02-14 21:02:18,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35810.97 MB 2025-02-14 21:02:18,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25398.61 MB 2025-02-14 21:02:18,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.89 MB 2025-02-14 21:02:18,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:02:18,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:02:18,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:02:18,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:18,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30455.72 MB 2025-02-14 21:02:18,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22571.43 MB 2025-02-14 21:02:18,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7884.29 MB 2025-02-14 21:02:18,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35810.97 MB 2025-02-14 21:02:18,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35810.97 MB 2025-02-14 21:02:18,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:02:18,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30455.72 MB 2025-02-14 21:02:18,486 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 21:02:18,487 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:02:18,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:02:18,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:02:18,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:02:18,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:02:18,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22571.43 MB 2025-02-14 21:02:18,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30989.58 MB 2025-02-14 21:02:18,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 21:02:18,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35810.97 MB 2025-02-14 21:02:18,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44180.70 MB 2025-02-14 21:02:18,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 21:02:18,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30989.58 MB 2025-02-14 21:02:18,649 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 21:02:18,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:02:18,650 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:02:18,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:02:18,651 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:02:18,656 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:02:18,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:02:18,657 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:02:18,657 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:03:27,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:03:27,634 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:03:27,639 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:03:27,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:03:27,642 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2079, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:03:27,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:03:27,643 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2079, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:03:59,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:03:59,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:03:59,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.99 seconds 2025-02-14 21:03:59,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:03:59,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27455.51 MB 2025-02-14 21:03:59,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34812.98 MB 2025-02-14 21:03:59,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7357.46 MB 2025-02-14 21:03:59,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56734.25 MB 2025-02-14 21:03:59,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38015.07 MB 2025-02-14 21:03:59,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18719.18 MB 2025-02-14 21:03:59,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43722.46 MB 2025-02-14 21:03:59,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:03:59,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:03:59,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:03:59,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:03:59,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.98 MB 2025-02-14 21:03:59,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26586.97 MB 2025-02-14 21:03:59,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8226.00 MB 2025-02-14 21:03:59,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38015.07 MB 2025-02-14 21:03:59,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55985.57 MB 2025-02-14 21:03:59,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17970.50 MB 2025-02-14 21:03:59,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47042.98 MB 2025-02-14 21:04:01,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:04:01,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:04:01,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 21:04:01,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:01,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26586.97 MB 2025-02-14 21:04:01,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27117.81 MB 2025-02-14 21:04:01,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:04:01,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55985.57 MB 2025-02-14 21:04:01,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29305.60 MB 2025-02-14 21:04:01,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26679.97 MB 2025-02-14 21:04:01,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31097.40 MB 2025-02-14 21:04:01,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:04:01,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:04:01,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:04:01,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:01,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-14 21:04:01,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29007.35 MB 2025-02-14 21:04:01,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:04:01,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29305.60 MB 2025-02-14 21:04:01,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32136.76 MB 2025-02-14 21:04:01,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 21:04:01,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30424.78 MB 2025-02-14 21:04:01,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:04:01,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:04:01,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:04:01,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:01,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29007.35 MB 2025-02-14 21:04:01,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-14 21:04:01,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:04:01,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32136.76 MB 2025-02-14 21:04:01,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38742.79 MB 2025-02-14 21:04:01,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 21:04:01,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-14 21:04:01,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:04:01,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:04:01,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:04:01,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:01,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-14 21:04:01,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-14 21:04:01,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:04:01,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29305.60 MB 2025-02-14 21:04:01,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38742.79 MB 2025-02-14 21:04:01,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 21:04:01,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-14 21:04:02,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:04:02,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:04:02,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:04:02,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:02,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32782.75 MB 2025-02-14 21:04:02,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33549.75 MB 2025-02-14 21:04:02,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:04:02,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38742.79 MB 2025-02-14 21:04:02,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39158.02 MB 2025-02-14 21:04:02,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:04:02,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34257.54 MB 2025-02-14 21:04:02,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:04:02,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:04:02,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:04:02,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:02,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33962.64 MB 2025-02-14 21:04:02,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34188.46 MB 2025-02-14 21:04:02,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.83 MB 2025-02-14 21:04:02,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39158.02 MB 2025-02-14 21:04:02,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39158.02 MB 2025-02-14 21:04:02,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:04:02,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34411.40 MB 2025-02-14 21:04:02,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:04:02,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:04:02,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.46 seconds 2025-02-14 21:04:02,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:02,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20212.11 MB 2025-02-14 21:04:02,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34388.95 MB 2025-02-14 21:04:02,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14176.84 MB 2025-02-14 21:04:02,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56734.25 MB 2025-02-14 21:04:02,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39158.02 MB 2025-02-14 21:04:02,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17576.23 MB 2025-02-14 21:04:02,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34411.40 MB 2025-02-14 21:04:02,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:04:02,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:04:02,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:04:02,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:02,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34388.95 MB 2025-02-14 21:04:02,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25203.08 MB 2025-02-14 21:04:02,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9185.87 MB 2025-02-14 21:04:02,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39158.02 MB 2025-02-14 21:04:02,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39158.02 MB 2025-02-14 21:04:02,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:04:02,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36889.55 MB 2025-02-14 21:04:02,397 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-14 21:04:02,397 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:04:02,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:04:02,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:04:02,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:04:02,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:02,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25203.08 MB 2025-02-14 21:04:02,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33604.61 MB 2025-02-14 21:04:02,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-14 21:04:02,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39158.02 MB 2025-02-14 21:04:02,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47513.08 MB 2025-02-14 21:04:02,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 21:04:02,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33604.61 MB 2025-02-14 21:04:02,561 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-14 21:04:02,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:04:02,562 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:04:02,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:04:02,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:04:02,568 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:04:02,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:04:02,569 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:04:02,569 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:04:37,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:04:37,009 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:04:37,014 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:04:37,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:04:37,018 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1334, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:04:37,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:04:37,019 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1334, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:04:57,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:04:57,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:04:57,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.63 seconds 2025-02-14 21:04:57,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:57,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22264.23 MB 2025-02-14 21:04:57,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26985.18 MB 2025-02-14 21:04:57,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4720.95 MB 2025-02-14 21:04:57,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55868.13 MB 2025-02-14 21:04:57,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35372.66 MB 2025-02-14 21:04:57,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20495.47 MB 2025-02-14 21:04:57,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35812.47 MB 2025-02-14 21:04:57,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:04:57,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:04:57,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:04:57,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:57,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26985.18 MB 2025-02-14 21:04:57,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22712.90 MB 2025-02-14 21:04:57,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4272.28 MB 2025-02-14 21:04:57,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35372.66 MB 2025-02-14 21:04:57,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45977.96 MB 2025-02-14 21:04:57,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10605.30 MB 2025-02-14 21:04:57,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41045.24 MB 2025-02-14 21:04:59,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:04:59,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:04:59,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 21:04:59,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:59,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22712.90 MB 2025-02-14 21:04:59,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23243.74 MB 2025-02-14 21:04:59,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:04:59,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45977.96 MB 2025-02-14 21:04:59,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30649.88 MB 2025-02-14 21:04:59,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15328.08 MB 2025-02-14 21:04:59,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27222.29 MB 2025-02-14 21:04:59,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:04:59,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:04:59,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:04:59,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:59,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23243.74 MB 2025-02-14 21:04:59,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25133.28 MB 2025-02-14 21:04:59,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:04:59,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30649.88 MB 2025-02-14 21:04:59,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30649.88 MB 2025-02-14 21:04:59,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:04:59,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26550.71 MB 2025-02-14 21:04:59,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:04:59,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:04:59,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:04:59,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:59,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25133.28 MB 2025-02-14 21:04:59,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27375.13 MB 2025-02-14 21:04:59,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:04:59,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30649.88 MB 2025-02-14 21:04:59,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 21:04:59,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:04:59,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32919.41 MB 2025-02-14 21:04:59,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:04:59,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:04:59,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:04:59,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:04:59,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23243.74 MB 2025-02-14 21:04:59,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27375.13 MB 2025-02-14 21:04:59,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:04:59,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30649.88 MB 2025-02-14 21:04:59,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 21:04:59,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:04:59,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32919.41 MB 2025-02-14 21:05:00,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:05:00,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:05:00,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:05:00,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:00,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28908.67 MB 2025-02-14 21:05:00,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29675.68 MB 2025-02-14 21:05:00,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:05:00,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35368.47 MB 2025-02-14 21:05:00,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 21:05:00,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:05:00,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30383.46 MB 2025-02-14 21:05:00,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:05:00,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:05:00,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:05:00,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:00,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30088.57 MB 2025-02-14 21:05:00,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30317.13 MB 2025-02-14 21:05:00,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 21:05:00,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 21:05:00,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 21:05:00,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:05:00,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30541.40 MB 2025-02-14 21:05:00,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:05:00,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:05:00,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.05 seconds 2025-02-14 21:05:00,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:00,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17616.47 MB 2025-02-14 21:05:00,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30518.21 MB 2025-02-14 21:05:00,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12901.74 MB 2025-02-14 21:05:00,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55868.13 MB 2025-02-14 21:05:00,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 21:05:00,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20084.42 MB 2025-02-14 21:05:00,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30541.40 MB 2025-02-14 21:05:00,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:05:00,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:05:00,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:05:00,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:00,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30518.21 MB 2025-02-14 21:05:00,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22620.86 MB 2025-02-14 21:05:00,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7897.35 MB 2025-02-14 21:05:00,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 21:05:00,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 21:05:00,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:05:00,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33029.87 MB 2025-02-14 21:05:00,357 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:05:00,357 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:05:00,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:05:00,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:05:00,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:05:00,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:00,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22620.86 MB 2025-02-14 21:05:00,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31059.88 MB 2025-02-14 21:05:00,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:05:00,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 21:05:00,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44174.41 MB 2025-02-14 21:05:00,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:05:00,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31059.88 MB 2025-02-14 21:05:00,521 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:05:00,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:00,522 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:05:00,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:00,523 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:05:00,528 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:05:00,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:00,529 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:05:00,529 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:05:09,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:09,678 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:05:09,683 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:05:09,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:09,686 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 675, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:05:09,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:09,687 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 675, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:05:20,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:05:20,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:05:20,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.53 seconds 2025-02-14 21:05:20,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:20,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17672.21 MB 2025-02-14 21:05:20,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20061.00 MB 2025-02-14 21:05:20,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2388.79 MB 2025-02-14 21:05:20,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56759.42 MB 2025-02-14 21:05:20,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24689.77 MB 2025-02-14 21:05:20,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32069.65 MB 2025-02-14 21:05:20,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28955.52 MB 2025-02-14 21:05:20,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:05:20,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:05:20,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 21:05:20,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:20,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20061.00 MB 2025-02-14 21:05:20,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19286.97 MB 2025-02-14 21:05:20,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -774.04 MB 2025-02-14 21:05:20,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24689.77 MB 2025-02-14 21:05:20,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31484.54 MB 2025-02-14 21:05:20,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6794.77 MB 2025-02-14 21:05:20,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28258.18 MB 2025-02-14 21:05:22,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:05:22,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:05:22,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 21:05:22,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19286.97 MB 2025-02-14 21:05:22,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19817.81 MB 2025-02-14 21:05:22,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:05:22,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31484.54 MB 2025-02-14 21:05:22,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23714.59 MB 2025-02-14 21:05:22,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7769.95 MB 2025-02-14 21:05:22,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23797.39 MB 2025-02-14 21:05:22,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:05:22,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:05:22,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:05:22,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19817.81 MB 2025-02-14 21:05:22,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21707.34 MB 2025-02-14 21:05:22,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:05:22,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 21:05:22,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25602.03 MB 2025-02-14 21:05:22,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:05:22,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23124.77 MB 2025-02-14 21:05:22,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:05:22,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:05:22,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 21:05:22,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21707.34 MB 2025-02-14 21:05:22,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23949.20 MB 2025-02-14 21:05:22,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:05:22,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25602.03 MB 2025-02-14 21:05:22,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31736.20 MB 2025-02-14 21:05:22,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:05:22,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29493.48 MB 2025-02-14 21:05:22,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:05:22,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:05:22,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 21:05:22,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19817.81 MB 2025-02-14 21:05:22,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23949.20 MB 2025-02-14 21:05:22,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:05:22,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23714.59 MB 2025-02-14 21:05:22,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31736.20 MB 2025-02-14 21:05:22,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 21:05:22,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29493.48 MB 2025-02-14 21:05:22,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:05:22,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:05:22,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:05:22,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25482.74 MB 2025-02-14 21:05:22,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26249.74 MB 2025-02-14 21:05:22,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:05:22,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31736.20 MB 2025-02-14 21:05:22,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-14 21:05:22,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:05:22,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26957.53 MB 2025-02-14 21:05:22,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:05:22,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:05:22,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:05:22,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26662.63 MB 2025-02-14 21:05:22,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.91 MB 2025-02-14 21:05:22,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.28 MB 2025-02-14 21:05:22,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32151.44 MB 2025-02-14 21:05:22,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-14 21:05:22,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:05:22,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27086.08 MB 2025-02-14 21:05:22,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:05:22,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:05:22,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.92 seconds 2025-02-14 21:05:22,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15320.46 MB 2025-02-14 21:05:22,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27092.98 MB 2025-02-14 21:05:22,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11772.52 MB 2025-02-14 21:05:22,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56759.42 MB 2025-02-14 21:05:22,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-14 21:05:22,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24607.98 MB 2025-02-14 21:05:22,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27092.98 MB 2025-02-14 21:05:22,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:05:22,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:05:22,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:05:22,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27092.98 MB 2025-02-14 21:05:22,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20324.85 MB 2025-02-14 21:05:22,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6768.13 MB 2025-02-14 21:05:22,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32151.44 MB 2025-02-14 21:05:22,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32151.44 MB 2025-02-14 21:05:22,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:05:22,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29604.65 MB 2025-02-14 21:05:22,891 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:05:22,891 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:05:22,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:05:22,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:05:22,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:05:22,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:05:22,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20324.85 MB 2025-02-14 21:05:22,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28763.87 MB 2025-02-14 21:05:22,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:05:22,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32151.44 MB 2025-02-14 21:05:22,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42641.39 MB 2025-02-14 21:05:22,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 21:05:22,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28763.87 MB 2025-02-14 21:05:23,054 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:05:23,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:23,055 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:05:23,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:23,056 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:05:23,061 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:05:23,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:05:23,062 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:05:23,062 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:06:54,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:06:54,562 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:06:54,568 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:06:54,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:06:54,573 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 159, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:06:54,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:06:54,573 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 159, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:06:57,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:06:57,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:06:57,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.45 seconds 2025-02-14 21:06:57,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14076.64 MB 2025-02-14 21:06:57,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14639.34 MB 2025-02-14 21:06:57,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 562.69 MB 2025-02-14 21:06:57,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55226.40 MB 2025-02-14 21:06:57,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:06:57,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35695.62 MB 2025-02-14 21:06:57,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23548.02 MB 2025-02-14 21:06:57,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:06:57,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:06:57,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:06:57,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14639.34 MB 2025-02-14 21:06:57,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14701.27 MB 2025-02-14 21:06:57,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 61.93 MB 2025-02-14 21:06:57,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:06:57,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:06:57,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:06:57,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16504.43 MB 2025-02-14 21:06:57,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:06:57,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:06:57,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.63 seconds 2025-02-14 21:06:57,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14701.27 MB 2025-02-14 21:06:57,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14872.46 MB 2025-02-14 21:06:57,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 171.20 MB 2025-02-14 21:06:57,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:06:57,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:06:57,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:06:57,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18870.92 MB 2025-02-14 21:06:57,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:06:57,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:06:57,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:06:57,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14872.40 MB 2025-02-14 21:06:57,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15481.63 MB 2025-02-14 21:06:57,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 609.23 MB 2025-02-14 21:06:57,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:06:57,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:06:57,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:06:57,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15938.75 MB 2025-02-14 21:06:57,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:06:57,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:06:57,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:06:57,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15481.63 MB 2025-02-14 21:06:57,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16204.67 MB 2025-02-14 21:06:57,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 723.04 MB 2025-02-14 21:06:57,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:06:57,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:06:57,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:06:57,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17992.66 MB 2025-02-14 21:06:57,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:06:57,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:06:57,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:06:57,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14872.40 MB 2025-02-14 21:06:57,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16204.67 MB 2025-02-14 21:06:57,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1332.27 MB 2025-02-14 21:06:57,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:06:57,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:06:57,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:06:57,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17992.66 MB 2025-02-14 21:06:57,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:06:57,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:06:57,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 21:06:57,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16699.24 MB 2025-02-14 21:06:57,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16946.89 MB 2025-02-14 21:06:57,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 247.65 MB 2025-02-14 21:06:57,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:06:57,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19660.80 MB 2025-02-14 21:06:57,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 130.02 MB 2025-02-14 21:06:57,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17187.01 MB 2025-02-14 21:06:57,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:06:57,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:06:57,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:06:57,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17080.05 MB 2025-02-14 21:06:57,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17283.95 MB 2025-02-14 21:06:57,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.89 MB 2025-02-14 21:06:57,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19660.80 MB 2025-02-14 21:06:57,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19660.80 MB 2025-02-14 21:06:57,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:06:57,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17283.95 MB 2025-02-14 21:06:57,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:06:57,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:06:57,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.24 seconds 2025-02-14 21:06:57,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:57,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13522.68 MB 2025-02-14 21:06:57,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17463.48 MB 2025-02-14 21:06:57,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3940.80 MB 2025-02-14 21:06:57,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55226.40 MB 2025-02-14 21:06:57,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19660.80 MB 2025-02-14 21:06:57,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35565.60 MB 2025-02-14 21:06:57,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17463.48 MB 2025-02-14 21:06:58,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:06:58,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:06:58,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 21:06:58,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:58,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17463.48 MB 2025-02-14 21:06:58,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16914.72 MB 2025-02-14 21:06:58,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -548.76 MB 2025-02-14 21:06:58,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19660.80 MB 2025-02-14 21:06:58,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19660.80 MB 2025-02-14 21:06:58,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:06:58,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18719.31 MB 2025-02-14 21:06:58,072 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7286, cut from 7288 2025-02-14 21:06:58,072 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:06:58,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:06:58,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:06:58,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:06:58,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:06:58,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16914.72 MB 2025-02-14 21:06:58,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24450.60 MB 2025-02-14 21:06:58,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7535.88 MB 2025-02-14 21:06:58,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19660.80 MB 2025-02-14 21:06:58,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29024.58 MB 2025-02-14 21:06:58,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9363.78 MB 2025-02-14 21:06:58,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24450.60 MB 2025-02-14 21:06:58,219 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7078] 2025-02-14 21:06:58,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:06:58,221 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:06:58,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:06:58,222 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:06:58,226 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:06:58,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:06:58,227 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:06:58,227 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:07:08,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:07:08,487 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:07:08,492 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:07:08,495 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:07:08,495 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2686, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:07:08,496 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:07:08,496 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2686, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:07:49,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:07:49,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:07:49,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.46 seconds 2025-02-14 21:07:49,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:49,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31685.18 MB 2025-02-14 21:07:49,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41191.57 MB 2025-02-14 21:07:49,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9506.39 MB 2025-02-14 21:07:49,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55238.98 MB 2025-02-14 21:07:49,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46340.77 MB 2025-02-14 21:07:49,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8898.22 MB 2025-02-14 21:07:49,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50697.18 MB 2025-02-14 21:07:50,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:07:50,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:07:50,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:07:50,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:50,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41191.57 MB 2025-02-14 21:07:50,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29741.53 MB 2025-02-14 21:07:50,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11450.05 MB 2025-02-14 21:07:50,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46340.77 MB 2025-02-14 21:07:50,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56906.22 MB 2025-02-14 21:07:50,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10565.45 MB 2025-02-14 21:07:50,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52517.32 MB 2025-02-14 21:07:52,032 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:07:52,032 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:07:52,032 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 21:07:52,032 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,032 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29741.53 MB 2025-02-14 21:07:52,032 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30272.37 MB 2025-02-14 21:07:52,032 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:07:52,032 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56906.22 MB 2025-02-14 21:07:52,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33495.71 MB 2025-02-14 21:07:52,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23410.51 MB 2025-02-14 21:07:52,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34251.95 MB 2025-02-14 21:07:52,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:07:52,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:07:52,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:07:52,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30272.37 MB 2025-02-14 21:07:52,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32161.81 MB 2025-02-14 21:07:52,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.44 MB 2025-02-14 21:07:52,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33495.71 MB 2025-02-14 21:07:52,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35385.25 MB 2025-02-14 21:07:52,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-14 21:07:52,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33579.24 MB 2025-02-14 21:07:52,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:07:52,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:07:52,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:07:52,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32161.81 MB 2025-02-14 21:07:52,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34403.67 MB 2025-02-14 21:07:52,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:07:52,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35385.25 MB 2025-02-14 21:07:52,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41519.42 MB 2025-02-14 21:07:52,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:07:52,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39947.95 MB 2025-02-14 21:07:52,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:07:52,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:07:52,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:07:52,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30272.37 MB 2025-02-14 21:07:52,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34403.67 MB 2025-02-14 21:07:52,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.30 MB 2025-02-14 21:07:52,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33495.71 MB 2025-02-14 21:07:52,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41519.42 MB 2025-02-14 21:07:52,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-14 21:07:52,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39947.95 MB 2025-02-14 21:07:52,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:07:52,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:07:52,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:07:52,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35937.21 MB 2025-02-14 21:07:52,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36704.21 MB 2025-02-14 21:07:52,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:07:52,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41519.42 MB 2025-02-14 21:07:52,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41936.75 MB 2025-02-14 21:07:52,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:07:52,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37412.00 MB 2025-02-14 21:07:52,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:07:52,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:07:52,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:07:52,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37117.10 MB 2025-02-14 21:07:52,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37343.35 MB 2025-02-14 21:07:52,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.25 MB 2025-02-14 21:07:52,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41936.75 MB 2025-02-14 21:07:52,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41936.75 MB 2025-02-14 21:07:52,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:07:52,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37564.22 MB 2025-02-14 21:07:52,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:07:52,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:07:52,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.94 seconds 2025-02-14 21:07:52,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22326.95 MB 2025-02-14 21:07:52,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37544.42 MB 2025-02-14 21:07:52,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15217.48 MB 2025-02-14 21:07:52,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45877.30 MB 2025-02-14 21:07:52,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41936.75 MB 2025-02-14 21:07:52,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3940.55 MB 2025-02-14 21:07:52,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37564.22 MB 2025-02-14 21:07:52,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:07:52,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:07:52,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:07:52,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37544.42 MB 2025-02-14 21:07:52,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27331.24 MB 2025-02-14 21:07:52,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10213.18 MB 2025-02-14 21:07:52,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41936.75 MB 2025-02-14 21:07:52,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41936.75 MB 2025-02-14 21:07:52,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:07:52,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40056.09 MB 2025-02-14 21:07:52,725 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:07:52,726 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:07:52,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:07:52,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:07:52,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:07:52,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:07:52,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27331.24 MB 2025-02-14 21:07:52,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35770.27 MB 2025-02-14 21:07:52,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:07:52,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41936.75 MB 2025-02-14 21:07:52,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46131.05 MB 2025-02-14 21:07:52,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 21:07:52,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35770.27 MB 2025-02-14 21:07:52,888 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:07:52,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:07:52,889 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:07:52,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:07:52,890 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:07:52,895 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:07:52,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:07:52,896 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:07:52,896 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:09:12,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:12,216 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:09:12,224 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:09:12,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:12,232 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:09:12,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:12,234 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:09:15,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:09:15,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:09:15,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.62 seconds 2025-02-14 21:09:15,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-14 21:09:15,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15395.85 MB 2025-02-14 21:09:15,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.50 MB 2025-02-14 21:09:15,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58716.06 MB 2025-02-14 21:09:15,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 21:09:15,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35911.63 MB 2025-02-14 21:09:15,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24276.21 MB 2025-02-14 21:09:15,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:09:15,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:09:15,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:09:15,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15395.85 MB 2025-02-14 21:09:15,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14275.41 MB 2025-02-14 21:09:15,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1120.44 MB 2025-02-14 21:09:15,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 21:09:15,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22804.43 MB 2025-02-14 21:09:15,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:15,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15631.83 MB 2025-02-14 21:09:15,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:09:15,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:09:15,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:09:15,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14275.41 MB 2025-02-14 21:09:15,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14295.31 MB 2025-02-14 21:09:15,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.91 MB 2025-02-14 21:09:15,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22804.43 MB 2025-02-14 21:09:15,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:09:15,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3275.75 MB 2025-02-14 21:09:15,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15233.29 MB 2025-02-14 21:09:15,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:09:15,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:09:15,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:09:15,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14295.25 MB 2025-02-14 21:09:15,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14366.09 MB 2025-02-14 21:09:15,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 70.84 MB 2025-02-14 21:09:15,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:09:15,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:09:15,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:15,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14419.25 MB 2025-02-14 21:09:15,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:09:15,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:09:15,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:09:15,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14366.09 MB 2025-02-14 21:09:15,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14450.27 MB 2025-02-14 21:09:15,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 84.18 MB 2025-02-14 21:09:15,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:09:15,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:09:15,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:15,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14658.91 MB 2025-02-14 21:09:15,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:09:15,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:09:15,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:09:15,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14295.25 MB 2025-02-14 21:09:15,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14450.27 MB 2025-02-14 21:09:15,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 155.02 MB 2025-02-14 21:09:15,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:09:15,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:09:15,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:15,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14658.91 MB 2025-02-14 21:09:15,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:09:15,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:09:15,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:09:15,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14507.78 MB 2025-02-14 21:09:15,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14537.33 MB 2025-02-14 21:09:15,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 29.55 MB 2025-02-14 21:09:15,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:09:15,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19541.26 MB 2025-02-14 21:09:15,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12.58 MB 2025-02-14 21:09:15,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14573.93 MB 2025-02-14 21:09:15,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:09:15,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:09:15,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:09:15,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14552.83 MB 2025-02-14 21:09:15,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14573.28 MB 2025-02-14 21:09:15,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 20.45 MB 2025-02-14 21:09:15,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19541.26 MB 2025-02-14 21:09:15,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19541.26 MB 2025-02-14 21:09:15,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:15,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14573.28 MB 2025-02-14 21:09:15,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:09:15,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:09:15,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.75 seconds 2025-02-14 21:09:15,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:15,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13773.53 MB 2025-02-14 21:09:15,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14610.09 MB 2025-02-14 21:09:15,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 836.56 MB 2025-02-14 21:09:15,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58716.06 MB 2025-02-14 21:09:15,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19541.26 MB 2025-02-14 21:09:15,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39174.80 MB 2025-02-14 21:09:15,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14610.09 MB 2025-02-14 21:09:16,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:09:16,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:09:16,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 21:09:16,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:16,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14610.09 MB 2025-02-14 21:09:16,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15161.95 MB 2025-02-14 21:09:16,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 551.86 MB 2025-02-14 21:09:16,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19541.26 MB 2025-02-14 21:09:16,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19545.46 MB 2025-02-14 21:09:16,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 21:09:16,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15217.13 MB 2025-02-14 21:09:16,056 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1483, cut from 1485 2025-02-14 21:09:16,057 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:09:16,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:09:16,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:09:16,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:09:16,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:16,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14417.41 MB 2025-02-14 21:09:16,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15962.09 MB 2025-02-14 21:09:16,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1544.68 MB 2025-02-14 21:09:16,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19545.46 MB 2025-02-14 21:09:16,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19545.46 MB 2025-02-14 21:09:16,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:16,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15962.09 MB 2025-02-14 21:09:16,089 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1275] 2025-02-14 21:09:16,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:16,091 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:09:16,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:16,092 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:09:16,096 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:09:16,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:16,097 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:09:16,097 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:09:30,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:30,195 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:09:30,200 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:09:30,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:30,204 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1666, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:09:30,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:30,205 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1666, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:09:56,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:09:56,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:09:56,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.81 seconds 2025-02-14 21:09:56,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:56,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24578.32 MB 2025-02-14 21:09:56,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30474.20 MB 2025-02-14 21:09:56,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5895.88 MB 2025-02-14 21:09:56,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28210.89 MB 2025-02-14 21:09:56,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32675.73 MB 2025-02-14 21:09:56,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4464.84 MB 2025-02-14 21:09:56,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39486.31 MB 2025-02-14 21:09:56,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:09:56,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:09:56,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:09:56,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:56,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30474.20 MB 2025-02-14 21:09:56,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24439.91 MB 2025-02-14 21:09:56,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6034.28 MB 2025-02-14 21:09:56,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32675.73 MB 2025-02-14 21:09:56,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55985.57 MB 2025-02-14 21:09:56,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23309.84 MB 2025-02-14 21:09:56,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46211.67 MB 2025-02-14 21:09:58,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:09:58,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:09:58,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 21:09:58,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24439.91 MB 2025-02-14 21:09:58,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24970.76 MB 2025-02-14 21:09:58,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:09:58,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55985.57 MB 2025-02-14 21:09:58,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26487.03 MB 2025-02-14 21:09:58,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29498.54 MB 2025-02-14 21:09:58,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28951.38 MB 2025-02-14 21:09:58,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:09:58,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:09:58,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:09:58,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24970.76 MB 2025-02-14 21:09:58,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26860.03 MB 2025-02-14 21:09:58,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-14 21:09:58,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26487.03 MB 2025-02-14 21:09:58,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29790.04 MB 2025-02-14 21:09:58,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 21:09:58,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28277.46 MB 2025-02-14 21:09:58,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:09:58,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:09:58,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:09:58,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26860.03 MB 2025-02-14 21:09:58,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29101.88 MB 2025-02-14 21:09:58,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:09:58,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29790.04 MB 2025-02-14 21:09:58,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36396.07 MB 2025-02-14 21:09:58,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 21:09:58,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34646.16 MB 2025-02-14 21:09:58,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:09:58,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:09:58,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:09:58,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24970.76 MB 2025-02-14 21:09:58,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29101.88 MB 2025-02-14 21:09:58,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-14 21:09:58,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26487.03 MB 2025-02-14 21:09:58,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36396.07 MB 2025-02-14 21:09:58,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 21:09:58,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34646.16 MB 2025-02-14 21:09:58,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:09:58,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:09:58,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 21:09:58,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30635.43 MB 2025-02-14 21:09:58,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31402.43 MB 2025-02-14 21:09:58,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:09:58,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36396.07 MB 2025-02-14 21:09:58,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36813.41 MB 2025-02-14 21:09:58,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:09:58,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32110.22 MB 2025-02-14 21:09:58,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:09:58,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:09:58,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:09:58,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31815.32 MB 2025-02-14 21:09:58,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32044.36 MB 2025-02-14 21:09:58,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.04 MB 2025-02-14 21:09:58,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36813.41 MB 2025-02-14 21:09:58,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36813.41 MB 2025-02-14 21:09:58,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:58,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32255.24 MB 2025-02-14 21:09:58,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:09:58,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:09:58,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.30 seconds 2025-02-14 21:09:58,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.18 MB 2025-02-14 21:09:58,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32245.43 MB 2025-02-14 21:09:58,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13472.25 MB 2025-02-14 21:09:58,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22403.87 MB 2025-02-14 21:09:58,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36813.41 MB 2025-02-14 21:09:58,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14409.53 MB 2025-02-14 21:09:58,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32255.24 MB 2025-02-14 21:09:58,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:09:58,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:09:58,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:09:58,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32245.43 MB 2025-02-14 21:09:58,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23777.36 MB 2025-02-14 21:09:58,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8468.07 MB 2025-02-14 21:09:58,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36813.41 MB 2025-02-14 21:09:58,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36813.41 MB 2025-02-14 21:09:58,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:09:58,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34757.10 MB 2025-02-14 21:09:58,789 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:09:58,789 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:09:58,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:09:58,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:09:58,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:09:58,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:09:58,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23777.36 MB 2025-02-14 21:09:58,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32216.38 MB 2025-02-14 21:09:58,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:09:58,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36813.41 MB 2025-02-14 21:09:58,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45204.11 MB 2025-02-14 21:09:58,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:09:58,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32216.38 MB 2025-02-14 21:09:58,958 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:09:58,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:58,959 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:09:58,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:58,960 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:09:58,965 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:09:58,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:09:58,966 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:09:58,966 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:10:08,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:10:08,861 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:10:08,866 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:10:08,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:10:08,869 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:10:08,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:10:08,870 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:10:12,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:10:12,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:10:12,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.03 seconds 2025-02-14 21:10:12,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:12,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14759.52 MB 2025-02-14 21:10:12,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15669.03 MB 2025-02-14 21:10:12,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 909.51 MB 2025-02-14 21:10:12,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57789.12 MB 2025-02-14 21:10:12,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23513.27 MB 2025-02-14 21:10:12,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34275.85 MB 2025-02-14 21:10:12,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24683.88 MB 2025-02-14 21:10:12,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:10:12,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:10:12,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:10:12,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:12,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15669.03 MB 2025-02-14 21:10:12,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15695.26 MB 2025-02-14 21:10:12,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 26.23 MB 2025-02-14 21:10:12,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23513.27 MB 2025-02-14 21:10:12,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23513.27 MB 2025-02-14 21:10:12,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:10:12,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18454.57 MB 2025-02-14 21:10:13,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:10:13,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:10:13,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.96 seconds 2025-02-14 21:10:13,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:13,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15695.26 MB 2025-02-14 21:10:13,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15958.03 MB 2025-02-14 21:10:13,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.77 MB 2025-02-14 21:10:13,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23513.27 MB 2025-02-14 21:10:13,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23041.41 MB 2025-02-14 21:10:13,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 21:10:13,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19949.85 MB 2025-02-14 21:10:13,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:10:13,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:10:13,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:10:13,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:13,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15958.03 MB 2025-02-14 21:10:13,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16893.12 MB 2025-02-14 21:10:13,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 935.09 MB 2025-02-14 21:10:13,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23041.41 MB 2025-02-14 21:10:13,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23041.41 MB 2025-02-14 21:10:13,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:10:13,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17594.75 MB 2025-02-14 21:10:13,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:10:13,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:10:13,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:10:13,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:13,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16893.12 MB 2025-02-14 21:10:13,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18002.87 MB 2025-02-14 21:10:13,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.75 MB 2025-02-14 21:10:13,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23041.41 MB 2025-02-14 21:10:13,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23041.41 MB 2025-02-14 21:10:13,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:10:13,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20747.26 MB 2025-02-14 21:10:13,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:10:13,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:10:13,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:10:13,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:13,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15958.03 MB 2025-02-14 21:10:13,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18002.87 MB 2025-02-14 21:10:13,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2044.84 MB 2025-02-14 21:10:13,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23041.41 MB 2025-02-14 21:10:13,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23041.41 MB 2025-02-14 21:10:13,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:10:13,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20747.26 MB 2025-02-14 21:10:14,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:10:14,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:10:14,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:10:14,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:14,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18761.98 MB 2025-02-14 21:10:14,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19142.59 MB 2025-02-14 21:10:14,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 380.62 MB 2025-02-14 21:10:14,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23041.41 MB 2025-02-14 21:10:14,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23246.93 MB 2025-02-14 21:10:14,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 205.52 MB 2025-02-14 21:10:14,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19496.25 MB 2025-02-14 21:10:14,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:10:14,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:10:14,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:10:14,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:14,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19346.98 MB 2025-02-14 21:10:14,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19566.70 MB 2025-02-14 21:10:14,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.72 MB 2025-02-14 21:10:14,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23246.93 MB 2025-02-14 21:10:14,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23246.93 MB 2025-02-14 21:10:14,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:10:14,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19616.86 MB 2025-02-14 21:10:14,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:10:14,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:10:14,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.21 seconds 2025-02-14 21:10:14,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:14,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13864.11 MB 2025-02-14 21:10:14,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19767.65 MB 2025-02-14 21:10:14,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5903.53 MB 2025-02-14 21:10:14,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57789.12 MB 2025-02-14 21:10:14,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23246.93 MB 2025-02-14 21:10:14,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34542.19 MB 2025-02-14 21:10:14,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19767.65 MB 2025-02-14 21:10:14,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:10:14,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:10:14,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:10:14,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:14,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14902.06 MB 2025-02-14 21:10:14,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17914.25 MB 2025-02-14 21:10:14,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.19 MB 2025-02-14 21:10:14,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23246.93 MB 2025-02-14 21:10:14,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23246.93 MB 2025-02-14 21:10:14,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:10:14,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18215.44 MB 2025-02-14 21:10:14,366 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 21:10:14,367 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:10:14,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:10:14,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:10:14,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:10:14,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:10:14,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17914.25 MB 2025-02-14 21:10:14,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26348.87 MB 2025-02-14 21:10:14,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 21:10:14,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23246.93 MB 2025-02-14 21:10:14,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31631.34 MB 2025-02-14 21:10:14,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 21:10:14,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26348.87 MB 2025-02-14 21:10:14,529 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 21:10:14,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:10:14,530 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:10:14,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:10:14,531 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:10:14,536 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:10:14,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:10:14,537 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:10:14,537 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:11:35,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:11:35,177 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:11:35,182 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:11:35,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:11:35,186 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 193, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:11:35,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:11:35,187 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 193, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:11:38,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:11:38,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:11:38,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.98 seconds 2025-02-14 21:11:38,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:38,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.56 MB 2025-02-14 21:11:38,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14996.58 MB 2025-02-14 21:11:38,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 683.02 MB 2025-02-14 21:11:38,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40015.76 MB 2025-02-14 21:11:38,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23041.41 MB 2025-02-14 21:11:38,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16974.35 MB 2025-02-14 21:11:38,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24011.42 MB 2025-02-14 21:11:38,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:11:38,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:11:38,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:11:38,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:38,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.58 MB 2025-02-14 21:11:38,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15327.50 MB 2025-02-14 21:11:38,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 330.92 MB 2025-02-14 21:11:38,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23041.41 MB 2025-02-14 21:11:38,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23041.41 MB 2025-02-14 21:11:38,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:11:38,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17711.45 MB 2025-02-14 21:11:39,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:11:39,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:11:39,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.92 seconds 2025-02-14 21:11:39,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15327.50 MB 2025-02-14 21:11:39,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15583.63 MB 2025-02-14 21:11:39,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 21:11:39,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23041.41 MB 2025-02-14 21:11:39,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22569.55 MB 2025-02-14 21:11:39,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 21:11:39,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19582.08 MB 2025-02-14 21:11:39,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:11:39,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:11:39,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:11:39,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15583.56 MB 2025-02-14 21:11:39,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16495.04 MB 2025-02-14 21:11:39,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 21:11:39,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22569.55 MB 2025-02-14 21:11:39,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22569.55 MB 2025-02-14 21:11:39,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:11:39,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17178.96 MB 2025-02-14 21:11:39,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:11:39,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:11:39,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:11:39,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16495.04 MB 2025-02-14 21:11:39,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17576.77 MB 2025-02-14 21:11:39,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-14 21:11:39,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22569.55 MB 2025-02-14 21:11:39,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22569.55 MB 2025-02-14 21:11:39,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:11:39,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20251.85 MB 2025-02-14 21:11:39,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:11:39,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:11:39,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:11:39,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15583.56 MB 2025-02-14 21:11:39,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17576.77 MB 2025-02-14 21:11:39,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-14 21:11:39,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22569.55 MB 2025-02-14 21:11:39,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22569.55 MB 2025-02-14 21:11:39,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:11:39,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20251.85 MB 2025-02-14 21:11:39,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:11:39,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:11:39,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:11:39,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18316.70 MB 2025-02-14 21:11:39,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18686.78 MB 2025-02-14 21:11:39,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-14 21:11:39,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22569.55 MB 2025-02-14 21:11:39,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22768.78 MB 2025-02-14 21:11:39,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 21:11:39,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19031.83 MB 2025-02-14 21:11:39,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:11:39,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:11:39,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:11:39,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18886.01 MB 2025-02-14 21:11:39,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19116.00 MB 2025-02-14 21:11:39,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.99 MB 2025-02-14 21:11:39,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22768.78 MB 2025-02-14 21:11:39,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22768.78 MB 2025-02-14 21:11:39,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:11:39,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19159.90 MB 2025-02-14 21:11:39,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:11:39,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:11:39,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.13 seconds 2025-02-14 21:11:39,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13641.13 MB 2025-02-14 21:11:39,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19317.07 MB 2025-02-14 21:11:39,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5675.94 MB 2025-02-14 21:11:39,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40015.76 MB 2025-02-14 21:11:39,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22770.88 MB 2025-02-14 21:11:39,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17244.88 MB 2025-02-14 21:11:39,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19317.07 MB 2025-02-14 21:11:39,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:11:39,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:11:39,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:11:39,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19317.07 MB 2025-02-14 21:11:39,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17668.63 MB 2025-02-14 21:11:39,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1648.44 MB 2025-02-14 21:11:39,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22770.88 MB 2025-02-14 21:11:39,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22770.88 MB 2025-02-14 21:11:39,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:11:39,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19317.07 MB 2025-02-14 21:11:39,600 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:11:39,600 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 21:11:39,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:11:39,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:11:39,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:11:39,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:11:39,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17668.63 MB 2025-02-14 21:11:39,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26107.65 MB 2025-02-14 21:11:39,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:11:39,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22770.88 MB 2025-02-14 21:11:39,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31161.58 MB 2025-02-14 21:11:39,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:11:39,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26107.65 MB 2025-02-14 21:11:39,769 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:11:39,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:11:39,770 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:11:39,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:11:39,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:11:39,776 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:11:39,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:11:39,777 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:11:39,777 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 21:12:41,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:12:41,864 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:12:41,869 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:12:41,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:12:41,873 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1737, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:12:41,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:12:41,874 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1737, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:13:08,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:13:08,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:13:08,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.59 seconds 2025-02-14 21:13:08,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:08,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25072.40 MB 2025-02-14 21:13:08,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31219.55 MB 2025-02-14 21:13:08,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6147.15 MB 2025-02-14 21:13:08,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43746.59 MB 2025-02-14 21:13:08,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35924.21 MB 2025-02-14 21:13:08,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7822.38 MB 2025-02-14 21:13:08,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40206.89 MB 2025-02-14 21:13:08,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:13:08,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:13:08,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:13:08,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:08,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31219.55 MB 2025-02-14 21:13:08,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24807.97 MB 2025-02-14 21:13:08,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6411.57 MB 2025-02-14 21:13:08,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35924.21 MB 2025-02-14 21:13:08,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57023.66 MB 2025-02-14 21:13:08,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21099.45 MB 2025-02-14 21:13:08,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48229.68 MB 2025-02-14 21:13:10,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:13:10,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:13:10,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:13:10,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:10,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24807.97 MB 2025-02-14 21:13:10,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25338.81 MB 2025-02-14 21:13:10,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:13:10,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57023.66 MB 2025-02-14 21:13:10,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27529.31 MB 2025-02-14 21:13:10,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29494.35 MB 2025-02-14 21:13:10,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29318.40 MB 2025-02-14 21:13:10,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:13:10,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:13:10,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:13:10,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:10,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.81 MB 2025-02-14 21:13:10,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27228.09 MB 2025-02-14 21:13:10,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-14 21:13:10,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27529.31 MB 2025-02-14 21:13:10,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30360.47 MB 2025-02-14 21:13:10,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 21:13:10,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28645.51 MB 2025-02-14 21:13:10,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:13:10,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:13:10,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 21:13:10,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:10,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27228.09 MB 2025-02-14 21:13:10,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29469.94 MB 2025-02-14 21:13:10,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:13:10,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30360.47 MB 2025-02-14 21:13:10,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36496.74 MB 2025-02-14 21:13:10,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 21:13:10,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35014.22 MB 2025-02-14 21:13:10,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:13:10,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:13:10,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 21:13:10,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:10,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.81 MB 2025-02-14 21:13:10,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29469.94 MB 2025-02-14 21:13:10,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-14 21:13:10,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27529.31 MB 2025-02-14 21:13:10,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36496.74 MB 2025-02-14 21:13:10,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8967.42 MB 2025-02-14 21:13:10,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35014.22 MB 2025-02-14 21:13:10,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:13:10,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:13:10,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:13:10,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:10,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31003.48 MB 2025-02-14 21:13:10,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31770.49 MB 2025-02-14 21:13:10,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:13:10,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36496.74 MB 2025-02-14 21:13:10,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36914.07 MB 2025-02-14 21:13:10,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:13:10,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32478.27 MB 2025-02-14 21:13:10,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:13:10,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:13:10,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:13:10,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:10,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32183.37 MB 2025-02-14 21:13:10,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32411.43 MB 2025-02-14 21:13:10,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-14 21:13:10,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36914.07 MB 2025-02-14 21:13:10,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36914.07 MB 2025-02-14 21:13:10,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:13:10,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32626.82 MB 2025-02-14 21:13:10,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:13:10,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:13:10,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.07 seconds 2025-02-14 21:13:10,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:10,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19020.55 MB 2025-02-14 21:13:10,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32611.66 MB 2025-02-14 21:13:10,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13591.11 MB 2025-02-14 21:13:10,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43746.59 MB 2025-02-14 21:13:10,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36914.07 MB 2025-02-14 21:13:10,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6832.52 MB 2025-02-14 21:13:10,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32626.82 MB 2025-02-14 21:13:11,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:13:11,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:13:11,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:13:11,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:11,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32611.66 MB 2025-02-14 21:13:11,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24012.24 MB 2025-02-14 21:13:11,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8599.43 MB 2025-02-14 21:13:11,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36914.07 MB 2025-02-14 21:13:11,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36914.07 MB 2025-02-14 21:13:11,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:13:11,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35112.89 MB 2025-02-14 21:13:11,234 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 21:13:11,234 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 21:13:11,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:13:11,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:13:11,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:13:11,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:11,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24012.24 MB 2025-02-14 21:13:11,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32417.32 MB 2025-02-14 21:13:11,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-14 21:13:11,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36914.07 MB 2025-02-14 21:13:11,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45269.12 MB 2025-02-14 21:13:11,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 21:13:11,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32417.32 MB 2025-02-14 21:13:11,402 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 21:13:11,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:11,404 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:13:11,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:11,405 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:13:11,409 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:13:11,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:11,410 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:13:11,411 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 21:13:26,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:26,505 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:13:26,510 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:13:26,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:26,513 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:13:26,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:26,514 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:13:46,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:13:46,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:13:46,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.97 seconds 2025-02-14 21:13:46,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:46,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21957.63 MB 2025-02-14 21:13:46,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26523.13 MB 2025-02-14 21:13:46,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4565.50 MB 2025-02-14 21:13:46,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53624.18 MB 2025-02-14 21:13:46,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34326.18 MB 2025-02-14 21:13:46,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19297.99 MB 2025-02-14 21:13:46,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35505.87 MB 2025-02-14 21:13:46,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:13:46,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:13:46,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:13:46,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:46,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26523.13 MB 2025-02-14 21:13:46,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22484.16 MB 2025-02-14 21:13:46,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4038.97 MB 2025-02-14 21:13:46,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34326.18 MB 2025-02-14 21:13:46,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42213.57 MB 2025-02-14 21:13:46,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7887.39 MB 2025-02-14 21:13:46,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38544.41 MB 2025-02-14 21:13:48,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:13:48,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:13:48,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 21:13:48,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:48,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.16 MB 2025-02-14 21:13:48,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23015.00 MB 2025-02-14 21:13:48,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:13:48,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42213.57 MB 2025-02-14 21:13:48,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29760.68 MB 2025-02-14 21:13:48,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12452.89 MB 2025-02-14 21:13:48,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.55 MB 2025-02-14 21:13:48,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:13:48,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:13:48,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:13:48,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:48,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-14 21:13:48,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24904.53 MB 2025-02-14 21:13:48,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:13:48,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29760.68 MB 2025-02-14 21:13:48,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29760.68 MB 2025-02-14 21:13:48,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:13:48,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26321.96 MB 2025-02-14 21:13:48,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:13:48,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:13:48,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:13:48,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:48,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24904.53 MB 2025-02-14 21:13:48,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-14 21:13:48,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:13:48,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29760.68 MB 2025-02-14 21:13:48,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34951.14 MB 2025-02-14 21:13:48,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:13:48,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-14 21:13:48,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:13:48,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:13:48,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:13:48,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:48,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-14 21:13:48,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-14 21:13:48,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:13:48,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29760.68 MB 2025-02-14 21:13:48,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34951.14 MB 2025-02-14 21:13:48,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:13:48,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-14 21:13:48,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:13:48,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:13:48,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:13:48,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:48,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28679.93 MB 2025-02-14 21:13:48,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29446.93 MB 2025-02-14 21:13:48,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:13:48,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34951.14 MB 2025-02-14 21:13:48,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 21:13:48,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:13:48,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30154.72 MB 2025-02-14 21:13:48,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:13:48,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:13:48,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:13:48,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:48,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29859.82 MB 2025-02-14 21:13:48,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30088.53 MB 2025-02-14 21:13:48,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.70 MB 2025-02-14 21:13:48,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35368.47 MB 2025-02-14 21:13:48,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 21:13:48,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:13:48,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30289.36 MB 2025-02-14 21:13:48,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:13:48,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:13:48,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.38 seconds 2025-02-14 21:13:48,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:48,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17463.17 MB 2025-02-14 21:13:48,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30288.39 MB 2025-02-14 21:13:48,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12825.22 MB 2025-02-14 21:13:48,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53624.18 MB 2025-02-14 21:13:48,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 21:13:48,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18255.71 MB 2025-02-14 21:13:48,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30289.36 MB 2025-02-14 21:13:49,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:13:49,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:13:49,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:13:49,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:49,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30288.39 MB 2025-02-14 21:13:49,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22449.51 MB 2025-02-14 21:13:49,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7838.89 MB 2025-02-14 21:13:49,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35368.47 MB 2025-02-14 21:13:49,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 21:13:49,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:13:49,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32785.01 MB 2025-02-14 21:13:49,179 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 21:13:49,179 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:13:49,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:13:49,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:13:49,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:13:49,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:13:49,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22449.51 MB 2025-02-14 21:13:49,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30837.92 MB 2025-02-14 21:13:49,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 21:13:49,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35368.47 MB 2025-02-14 21:13:49,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43708.84 MB 2025-02-14 21:13:49,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-14 21:13:49,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30837.92 MB 2025-02-14 21:13:49,341 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 21:13:49,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:49,342 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:13:49,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:49,343 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:13:49,348 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:13:49,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:13:49,349 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:13:49,349 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:14:09,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:09,310 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:14:09,314 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:14:09,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:09,318 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 352, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:14:09,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:09,319 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 352, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:14:14,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:14:14,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:14:14,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.48 seconds 2025-02-14 21:14:14,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:14,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15421.50 MB 2025-02-14 21:14:14,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16667.21 MB 2025-02-14 21:14:14,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1245.71 MB 2025-02-14 21:14:14,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56218.35 MB 2025-02-14 21:14:14,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21890.07 MB 2025-02-14 21:14:14,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34328.28 MB 2025-02-14 21:14:14,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25572.35 MB 2025-02-14 21:14:14,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:14:14,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:14:14,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:14:14,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:14,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16667.21 MB 2025-02-14 21:14:14,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17271.12 MB 2025-02-14 21:14:14,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 603.91 MB 2025-02-14 21:14:14,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21890.07 MB 2025-02-14 21:14:14,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24362.61 MB 2025-02-14 21:14:14,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2472.54 MB 2025-02-14 21:14:14,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21668.48 MB 2025-02-14 21:14:16,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:14:16,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:14:16,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.68 seconds 2025-02-14 21:14:16,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:16,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17271.12 MB 2025-02-14 21:14:16,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.26 MB 2025-02-14 21:14:16,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 467.14 MB 2025-02-14 21:14:16,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24362.61 MB 2025-02-14 21:14:16,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23418.90 MB 2025-02-14 21:14:16,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 21:14:16,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21695.58 MB 2025-02-14 21:14:16,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:14:16,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:14:16,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:14:16,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:16,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17738.26 MB 2025-02-14 21:14:16,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19401.30 MB 2025-02-14 21:14:16,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1663.04 MB 2025-02-14 21:14:16,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23418.90 MB 2025-02-14 21:14:16,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23418.90 MB 2025-02-14 21:14:16,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:14:16,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20648.64 MB 2025-02-14 21:14:16,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:14:16,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:14:16,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 21:14:16,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:16,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19401.30 MB 2025-02-14 21:14:16,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21374.14 MB 2025-02-14 21:14:16,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1972.84 MB 2025-02-14 21:14:16,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23418.90 MB 2025-02-14 21:14:16,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28401.73 MB 2025-02-14 21:14:16,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4982.83 MB 2025-02-14 21:14:16,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26253.10 MB 2025-02-14 21:14:16,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:14:16,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:14:16,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:14:16,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:16,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17738.26 MB 2025-02-14 21:14:16,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21374.14 MB 2025-02-14 21:14:16,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3635.88 MB 2025-02-14 21:14:16,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23418.90 MB 2025-02-14 21:14:16,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28401.73 MB 2025-02-14 21:14:16,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4982.83 MB 2025-02-14 21:14:16,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26253.10 MB 2025-02-14 21:14:16,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:14:16,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:14:16,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 21:14:16,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:16,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22723.66 MB 2025-02-14 21:14:16,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23398.62 MB 2025-02-14 21:14:16,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 674.96 MB 2025-02-14 21:14:16,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28401.73 MB 2025-02-14 21:14:16,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28766.63 MB 2025-02-14 21:14:16,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 364.90 MB 2025-02-14 21:14:16,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24021.48 MB 2025-02-14 21:14:16,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:14:16,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:14:16,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:14:16,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:16,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23761.97 MB 2025-02-14 21:14:16,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23987.87 MB 2025-02-14 21:14:16,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.91 MB 2025-02-14 21:14:16,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28766.63 MB 2025-02-14 21:14:16,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28766.63 MB 2025-02-14 21:14:16,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:14:16,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24164.15 MB 2025-02-14 21:14:16,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:14:16,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:14:16,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.54 seconds 2025-02-14 21:14:16,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:16,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14195.10 MB 2025-02-14 21:14:16,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24188.53 MB 2025-02-14 21:14:16,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9993.43 MB 2025-02-14 21:14:16,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56218.35 MB 2025-02-14 21:14:16,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28766.63 MB 2025-02-14 21:14:16,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27451.72 MB 2025-02-14 21:14:16,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24188.53 MB 2025-02-14 21:14:17,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:14:17,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:14:17,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:14:17,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:17,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24188.53 MB 2025-02-14 21:14:17,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18966.49 MB 2025-02-14 21:14:17,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5222.04 MB 2025-02-14 21:14:17,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28766.63 MB 2025-02-14 21:14:17,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28766.63 MB 2025-02-14 21:14:17,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:14:17,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27396.78 MB 2025-02-14 21:14:17,152 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 21:14:17,152 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:14:17,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:14:17,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:14:17,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:14:17,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:17,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18966.49 MB 2025-02-14 21:14:17,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27388.45 MB 2025-02-14 21:14:17,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 21:14:17,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28766.63 MB 2025-02-14 21:14:17,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37138.46 MB 2025-02-14 21:14:17,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 21:14:17,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27388.45 MB 2025-02-14 21:14:17,314 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 21:14:17,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:17,316 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:14:17,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:17,317 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:14:17,321 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:14:17,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:17,322 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:14:17,322 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:14:41,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:41,968 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:14:41,972 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:14:41,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:41,976 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 473, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:14:41,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:41,977 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 473, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:14:49,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:14:49,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:14:49,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.31 seconds 2025-02-14 21:14:49,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:49,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16264.65 MB 2025-02-14 21:14:49,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.57 MB 2025-02-14 21:14:49,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.92 MB 2025-02-14 21:14:49,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45510.30 MB 2025-02-14 21:14:49,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24941.43 MB 2025-02-14 21:14:49,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20568.87 MB 2025-02-14 21:14:49,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26868.48 MB 2025-02-14 21:14:49,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:14:49,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:14:49,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 21:14:49,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:49,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.57 MB 2025-02-14 21:14:49,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18237.88 MB 2025-02-14 21:14:49,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.31 MB 2025-02-14 21:14:49,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24941.43 MB 2025-02-14 21:14:49,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28481.42 MB 2025-02-14 21:14:49,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3539.99 MB 2025-02-14 21:14:49,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25384.71 MB 2025-02-14 21:14:51,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:14:51,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:14:51,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 21:14:51,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18237.88 MB 2025-02-14 21:14:51,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18768.72 MB 2025-02-14 21:14:51,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:14:51,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28481.42 MB 2025-02-14 21:14:51,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26122.13 MB 2025-02-14 21:14:51,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2359.30 MB 2025-02-14 21:14:51,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22747.27 MB 2025-02-14 21:14:51,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:14:51,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:14:51,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:14:51,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-14 21:14:51,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20658.25 MB 2025-02-14 21:14:51,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:14:51,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26122.13 MB 2025-02-14 21:14:51,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26122.13 MB 2025-02-14 21:14:51,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:14:51,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22075.68 MB 2025-02-14 21:14:51,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:14:51,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:14:51,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:14:51,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20658.25 MB 2025-02-14 21:14:51,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22901.16 MB 2025-02-14 21:14:51,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-14 21:14:51,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26122.13 MB 2025-02-14 21:14:51,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30605.84 MB 2025-02-14 21:14:51,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4483.71 MB 2025-02-14 21:14:51,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28445.44 MB 2025-02-14 21:14:51,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:14:51,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:14:51,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:14:51,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-14 21:14:51,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22901.16 MB 2025-02-14 21:14:51,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-14 21:14:51,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26122.13 MB 2025-02-14 21:14:51,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30605.84 MB 2025-02-14 21:14:51,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4483.71 MB 2025-02-14 21:14:51,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28445.44 MB 2025-02-14 21:14:51,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:14:51,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:14:51,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:14:51,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24434.70 MB 2025-02-14 21:14:51,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25201.70 MB 2025-02-14 21:14:51,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:14:51,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30605.84 MB 2025-02-14 21:14:51,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31018.98 MB 2025-02-14 21:14:51,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 21:14:51,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25909.49 MB 2025-02-14 21:14:51,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:14:51,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:14:51,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:14:51,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25614.59 MB 2025-02-14 21:14:51,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25848.48 MB 2025-02-14 21:14:51,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.89 MB 2025-02-14 21:14:51,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31018.98 MB 2025-02-14 21:14:51,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31018.98 MB 2025-02-14 21:14:51,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:14:51,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26031.57 MB 2025-02-14 21:14:51,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:14:51,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:14:51,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.65 seconds 2025-02-14 21:14:51,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14616.68 MB 2025-02-14 21:14:51,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26049.55 MB 2025-02-14 21:14:51,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11432.88 MB 2025-02-14 21:14:51,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45510.30 MB 2025-02-14 21:14:51,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31018.98 MB 2025-02-14 21:14:51,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14491.32 MB 2025-02-14 21:14:51,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26049.55 MB 2025-02-14 21:14:51,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:14:51,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:14:51,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:14:51,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26049.55 MB 2025-02-14 21:14:51,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19621.07 MB 2025-02-14 21:14:51,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6428.49 MB 2025-02-14 21:14:51,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31018.98 MB 2025-02-14 21:14:51,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31018.98 MB 2025-02-14 21:14:51,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:14:51,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28561.22 MB 2025-02-14 21:14:51,920 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:14:51,920 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:14:51,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:14:51,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:14:51,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:14:51,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:14:51,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19621.07 MB 2025-02-14 21:14:51,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28060.09 MB 2025-02-14 21:14:51,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:14:51,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31018.98 MB 2025-02-14 21:14:51,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39409.68 MB 2025-02-14 21:14:51,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:14:51,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28060.09 MB 2025-02-14 21:14:52,083 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:14:52,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:52,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:14:52,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:52,085 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:14:52,090 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:14:52,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:14:52,091 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:14:52,091 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:15:01,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:15:01,606 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:15:01,611 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:15:01,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:15:01,615 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 443, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:15:01,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:15:01,616 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 443, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:15:08,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:15:08,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:15:08,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.84 seconds 2025-02-14 21:15:08,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:08,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16055.60 MB 2025-02-14 21:15:08,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17623.35 MB 2025-02-14 21:15:08,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1567.75 MB 2025-02-14 21:15:08,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51994.69 MB 2025-02-14 21:15:08,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24941.43 MB 2025-02-14 21:15:08,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27053.26 MB 2025-02-14 21:15:08,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26432.94 MB 2025-02-14 21:15:08,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:15:08,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:15:08,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 21:15:08,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:08,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17623.35 MB 2025-02-14 21:15:08,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18080.87 MB 2025-02-14 21:15:08,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 457.52 MB 2025-02-14 21:15:08,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24941.43 MB 2025-02-14 21:15:08,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27772.58 MB 2025-02-14 21:15:08,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 21:15:08,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24998.27 MB 2025-02-14 21:15:10,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:15:10,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:15:10,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 21:15:10,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:10,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18080.87 MB 2025-02-14 21:15:10,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18611.71 MB 2025-02-14 21:15:10,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:15:10,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27772.58 MB 2025-02-14 21:15:10,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25526.53 MB 2025-02-14 21:15:10,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2246.05 MB 2025-02-14 21:15:10,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22590.26 MB 2025-02-14 21:15:10,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:15:10,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:15:10,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:15:10,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:10,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18611.71 MB 2025-02-14 21:15:10,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20501.24 MB 2025-02-14 21:15:10,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:15:10,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25526.53 MB 2025-02-14 21:15:10,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25526.53 MB 2025-02-14 21:15:10,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:15:10,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21918.67 MB 2025-02-14 21:15:10,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:15:10,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:15:10,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:15:10,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:10,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20501.24 MB 2025-02-14 21:15:10,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22743.10 MB 2025-02-14 21:15:10,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:15:10,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25526.53 MB 2025-02-14 21:15:10,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31188.84 MB 2025-02-14 21:15:10,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:15:10,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28287.38 MB 2025-02-14 21:15:10,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:15:10,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:15:10,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:15:10,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:10,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18611.71 MB 2025-02-14 21:15:10,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22743.10 MB 2025-02-14 21:15:10,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:15:10,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25526.53 MB 2025-02-14 21:15:10,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31188.84 MB 2025-02-14 21:15:10,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:15:10,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28287.38 MB 2025-02-14 21:15:10,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:15:10,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:15:10,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:15:10,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:10,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24276.64 MB 2025-02-14 21:15:10,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25043.64 MB 2025-02-14 21:15:10,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:15:10,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31188.84 MB 2025-02-14 21:15:10,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31601.98 MB 2025-02-14 21:15:10,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 21:15:10,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25751.43 MB 2025-02-14 21:15:10,826 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:15:10,826 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:15:10,826 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:15:10,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:10,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25456.53 MB 2025-02-14 21:15:10,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25685.22 MB 2025-02-14 21:15:10,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-14 21:15:10,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31601.98 MB 2025-02-14 21:15:10,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31601.98 MB 2025-02-14 21:15:10,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:15:10,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25892.47 MB 2025-02-14 21:15:10,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:15:10,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:15:10,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.21 seconds 2025-02-14 21:15:10,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:10,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14512.15 MB 2025-02-14 21:15:10,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25886.30 MB 2025-02-14 21:15:10,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11374.14 MB 2025-02-14 21:15:10,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51994.69 MB 2025-02-14 21:15:10,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31601.98 MB 2025-02-14 21:15:10,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20392.71 MB 2025-02-14 21:15:10,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25892.47 MB 2025-02-14 21:15:11,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:15:11,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:15:11,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:15:11,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:11,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25886.30 MB 2025-02-14 21:15:11,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19516.54 MB 2025-02-14 21:15:11,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6369.75 MB 2025-02-14 21:15:11,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31601.98 MB 2025-02-14 21:15:11,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31601.98 MB 2025-02-14 21:15:11,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:15:11,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28397.96 MB 2025-02-14 21:15:11,113 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:15:11,114 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:15:11,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:15:11,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:15:11,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:15:11,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:15:11,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19516.54 MB 2025-02-14 21:15:11,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27955.57 MB 2025-02-14 21:15:11,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:15:11,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31601.98 MB 2025-02-14 21:15:11,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39992.69 MB 2025-02-14 21:15:11,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:15:11,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27955.57 MB 2025-02-14 21:15:11,281 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:15:11,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:15:11,283 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:15:11,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:15:11,284 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:15:11,289 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:15:11,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:15:11,290 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:15:11,290 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:16:21,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:21,221 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:16:21,226 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:16:21,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:21,230 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:16:21,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:21,231 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:16:24,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:16:24,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:16:24,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.86 seconds 2025-02-14 21:16:24,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:24,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-14 21:16:24,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-14 21:16:24,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-14 21:16:24,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52577.70 MB 2025-02-14 21:16:24,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 21:16:24,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32575.06 MB 2025-02-14 21:16:24,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.15 MB 2025-02-14 21:16:24,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:16:24,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:16:24,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:16:24,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:24,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-14 21:16:24,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15178.74 MB 2025-02-14 21:16:24,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.71 MB 2025-02-14 21:16:24,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 21:16:24,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 21:16:24,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:16:24,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17444.63 MB 2025-02-14 21:16:24,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:16:24,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:16:24,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 21:16:24,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:24,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15178.74 MB 2025-02-14 21:16:24,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15413.64 MB 2025-02-14 21:16:24,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 21:16:24,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 21:16:24,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19933.43 MB 2025-02-14 21:16:24,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -69.21 MB 2025-02-14 21:16:24,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19349.43 MB 2025-02-14 21:16:24,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:16:24,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:16:24,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:16:24,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:24,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15413.57 MB 2025-02-14 21:16:24,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16249.49 MB 2025-02-14 21:16:24,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 21:16:24,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19933.43 MB 2025-02-14 21:16:24,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19933.43 MB 2025-02-14 21:16:24,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:16:24,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16876.70 MB 2025-02-14 21:16:25,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:16:25,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:16:25,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:16:25,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:25,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16249.49 MB 2025-02-14 21:16:25,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17241.54 MB 2025-02-14 21:16:25,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 21:16:25,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19933.43 MB 2025-02-14 21:16:25,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21191.72 MB 2025-02-14 21:16:25,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-14 21:16:25,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19694.85 MB 2025-02-14 21:16:25,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:16:25,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:16:25,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:16:25,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:25,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15413.57 MB 2025-02-14 21:16:25,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17241.54 MB 2025-02-14 21:16:25,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 21:16:25,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19933.43 MB 2025-02-14 21:16:25,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21191.72 MB 2025-02-14 21:16:25,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1258.29 MB 2025-02-14 21:16:25,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19694.85 MB 2025-02-14 21:16:25,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:16:25,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:16:25,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:16:25,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:25,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17920.14 MB 2025-02-14 21:16:25,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18259.53 MB 2025-02-14 21:16:25,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 21:16:25,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21191.72 MB 2025-02-14 21:16:25,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21374.17 MB 2025-02-14 21:16:25,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 21:16:25,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18578.33 MB 2025-02-14 21:16:25,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:16:25,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:16:25,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:16:25,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:25,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18442.24 MB 2025-02-14 21:16:25,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18665.09 MB 2025-02-14 21:16:25,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.85 MB 2025-02-14 21:16:25,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21374.17 MB 2025-02-14 21:16:25,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21374.17 MB 2025-02-14 21:16:25,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:16:25,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18695.25 MB 2025-02-14 21:16:25,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:16:25,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:16:25,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.91 seconds 2025-02-14 21:16:25,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:25,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-14 21:16:25,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18866.17 MB 2025-02-14 21:16:25,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5249.42 MB 2025-02-14 21:16:25,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52577.70 MB 2025-02-14 21:16:25,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21374.17 MB 2025-02-14 21:16:25,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31203.52 MB 2025-02-14 21:16:25,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18866.17 MB 2025-02-14 21:16:25,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:16:25,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:16:25,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:16:25,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:25,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18866.17 MB 2025-02-14 21:16:25,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17568.97 MB 2025-02-14 21:16:25,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1297.19 MB 2025-02-14 21:16:25,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21374.17 MB 2025-02-14 21:16:25,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21374.17 MB 2025-02-14 21:16:25,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:16:25,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19100.59 MB 2025-02-14 21:16:25,431 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:16:25,431 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:16:25,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:16:25,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:16:25,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:16:25,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:16:25,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17568.97 MB 2025-02-14 21:16:25,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.00 MB 2025-02-14 21:16:25,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:16:25,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21374.17 MB 2025-02-14 21:16:25,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29764.88 MB 2025-02-14 21:16:25,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:16:25,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26008.00 MB 2025-02-14 21:16:25,599 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:16:25,600 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:25,600 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:16:25,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:25,601 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:16:25,606 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:16:25,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:25,607 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:16:25,607 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:16:34,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:34,551 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:16:34,556 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:16:34,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:34,559 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1656, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:16:34,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:16:34,560 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1656, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:17:00,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:17:00,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:17:00,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.51 seconds 2025-02-14 21:17:00,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:00,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24507.98 MB 2025-02-14 21:17:00,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30369.52 MB 2025-02-14 21:17:00,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5861.54 MB 2025-02-14 21:17:00,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42349.89 MB 2025-02-14 21:17:00,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36559.65 MB 2025-02-14 21:17:00,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5790.24 MB 2025-02-14 21:17:00,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39189.48 MB 2025-02-14 21:17:00,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:17:00,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:17:00,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:17:00,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:00,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30369.52 MB 2025-02-14 21:17:00,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24386.88 MB 2025-02-14 21:17:00,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5982.64 MB 2025-02-14 21:17:00,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36559.65 MB 2025-02-14 21:17:00,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55750.69 MB 2025-02-14 21:17:00,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19191.04 MB 2025-02-14 21:17:00,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47099.06 MB 2025-02-14 21:17:02,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:17:02,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:17:02,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.07 seconds 2025-02-14 21:17:02,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24386.88 MB 2025-02-14 21:17:02,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24917.72 MB 2025-02-14 21:17:02,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:17:02,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55750.69 MB 2025-02-14 21:17:02,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27919.38 MB 2025-02-14 21:17:02,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27831.30 MB 2025-02-14 21:17:02,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28897.31 MB 2025-02-14 21:17:02,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:17:02,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:17:02,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:17:02,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24917.72 MB 2025-02-14 21:17:02,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26807.25 MB 2025-02-14 21:17:02,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:17:02,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27919.38 MB 2025-02-14 21:17:02,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29806.82 MB 2025-02-14 21:17:02,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:17:02,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28224.68 MB 2025-02-14 21:17:02,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:17:02,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:17:02,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:17:02,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26807.25 MB 2025-02-14 21:17:02,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29049.11 MB 2025-02-14 21:17:02,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:17:02,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29806.82 MB 2025-02-14 21:17:02,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-14 21:17:02,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 21:17:02,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34593.39 MB 2025-02-14 21:17:02,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:17:02,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:17:02,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:17:02,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24917.72 MB 2025-02-14 21:17:02,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29049.11 MB 2025-02-14 21:17:02,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:17:02,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27919.38 MB 2025-02-14 21:17:02,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36412.85 MB 2025-02-14 21:17:02,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 21:17:02,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34593.39 MB 2025-02-14 21:17:02,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:17:02,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:17:02,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:17:02,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30582.65 MB 2025-02-14 21:17:02,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31349.65 MB 2025-02-14 21:17:02,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:17:02,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36412.85 MB 2025-02-14 21:17:02,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 21:17:02,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:17:02,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32057.44 MB 2025-02-14 21:17:02,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:17:02,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:17:02,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:17:02,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31762.54 MB 2025-02-14 21:17:02,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31988.26 MB 2025-02-14 21:17:02,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.72 MB 2025-02-14 21:17:02,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36828.09 MB 2025-02-14 21:17:02,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 21:17:02,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:17:02,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32177.23 MB 2025-02-14 21:17:02,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:17:02,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:17:02,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.10 seconds 2025-02-14 21:17:02,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18738.34 MB 2025-02-14 21:17:02,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32188.74 MB 2025-02-14 21:17:02,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13450.40 MB 2025-02-14 21:17:02,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42349.89 MB 2025-02-14 21:17:02,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 21:17:02,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5521.80 MB 2025-02-14 21:17:02,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32188.74 MB 2025-02-14 21:17:02,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:17:02,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:17:02,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-14 21:17:02,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32188.74 MB 2025-02-14 21:17:02,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23727.53 MB 2025-02-14 21:17:02,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8461.21 MB 2025-02-14 21:17:02,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36828.09 MB 2025-02-14 21:17:02,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36828.09 MB 2025-02-14 21:17:02,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:17:02,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34687.81 MB 2025-02-14 21:17:02,978 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-14 21:17:02,978 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:17:02,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:17:02,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:17:02,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:17:02,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:17:02,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23727.53 MB 2025-02-14 21:17:02,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32124.30 MB 2025-02-14 21:17:02,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.77 MB 2025-02-14 21:17:02,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36828.09 MB 2025-02-14 21:17:02,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45176.85 MB 2025-02-14 21:17:02,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8348.76 MB 2025-02-14 21:17:02,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32124.30 MB 2025-02-14 21:17:03,143 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-14 21:17:03,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:17:03,144 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:17:03,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:17:03,145 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:17:03,150 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:17:03,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:17:03,151 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:17:03,151 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:18:15,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:15,351 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:18:15,356 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:18:15,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:15,360 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:18:15,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:15,361 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:18:18,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:18:18,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:18:18,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.80 seconds 2025-02-14 21:18:18,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:18,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-14 21:18:18,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14881.00 MB 2025-02-14 21:18:18,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-14 21:18:18,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57698.94 MB 2025-02-14 21:18:18,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 21:18:18,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36754.69 MB 2025-02-14 21:18:18,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23708.28 MB 2025-02-14 21:18:18,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:18:18,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:18:18,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:18:18,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:18,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14881.00 MB 2025-02-14 21:18:18,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15066.64 MB 2025-02-14 21:18:18,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 185.64 MB 2025-02-14 21:18:18,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 21:18:18,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 21:18:18,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:18,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17206.68 MB 2025-02-14 21:18:18,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:18:18,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:18:18,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 21:18:18,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:18,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15066.64 MB 2025-02-14 21:18:18,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15284.29 MB 2025-02-14 21:18:18,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.65 MB 2025-02-14 21:18:18,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 21:18:18,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 21:18:18,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:18,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19236.29 MB 2025-02-14 21:18:18,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:18:18,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:18:18,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:18:18,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:18,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15284.22 MB 2025-02-14 21:18:18,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16058.74 MB 2025-02-14 21:18:18,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 774.52 MB 2025-02-14 21:18:18,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 21:18:18,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 21:18:18,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:18,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16639.90 MB 2025-02-14 21:18:19,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:18:19,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:18:19,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:18:19,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:19,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16058.74 MB 2025-02-14 21:18:19,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16978.01 MB 2025-02-14 21:18:19,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 919.26 MB 2025-02-14 21:18:19,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 21:18:19,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 21:18:19,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:19,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19251.13 MB 2025-02-14 21:18:19,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:18:19,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:18:19,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 21:18:19,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:19,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15284.22 MB 2025-02-14 21:18:19,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16978.01 MB 2025-02-14 21:18:19,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1693.79 MB 2025-02-14 21:18:19,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 21:18:19,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 21:18:19,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:19,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19251.13 MB 2025-02-14 21:18:19,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:18:19,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:18:19,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:18:19,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:19,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17606.76 MB 2025-02-14 21:18:19,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17921.23 MB 2025-02-14 21:18:19,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 314.47 MB 2025-02-14 21:18:19,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 21:18:19,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21114.13 MB 2025-02-14 21:18:19,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-14 21:18:19,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18218.51 MB 2025-02-14 21:18:19,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:18:19,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:18:19,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:18:19,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:19,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18090.52 MB 2025-02-14 21:18:19,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18302.73 MB 2025-02-14 21:18:19,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.21 MB 2025-02-14 21:18:19,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21114.13 MB 2025-02-14 21:18:19,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21114.13 MB 2025-02-14 21:18:19,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:19,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18331.56 MB 2025-02-14 21:18:19,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:18:19,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:18:19,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.84 seconds 2025-02-14 21:18:19,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:19,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13602.81 MB 2025-02-14 21:18:19,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18503.39 MB 2025-02-14 21:18:19,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4900.58 MB 2025-02-14 21:18:19,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57698.94 MB 2025-02-14 21:18:19,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21114.13 MB 2025-02-14 21:18:19,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36584.82 MB 2025-02-14 21:18:19,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18503.39 MB 2025-02-14 21:18:19,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:18:19,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:18:19,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 21:18:19,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:19,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18503.39 MB 2025-02-14 21:18:19,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17486.97 MB 2025-02-14 21:18:19,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1016.42 MB 2025-02-14 21:18:19,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21114.13 MB 2025-02-14 21:18:19,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21114.13 MB 2025-02-14 21:18:19,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:19,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19205.19 MB 2025-02-14 21:18:19,513 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 21:18:19,514 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:18:19,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:18:19,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:18:19,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:18:19,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:19,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17486.97 MB 2025-02-14 21:18:19,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25908.93 MB 2025-02-14 21:18:19,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 21:18:19,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21114.13 MB 2025-02-14 21:18:19,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31578.91 MB 2025-02-14 21:18:19,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 21:18:19,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25908.93 MB 2025-02-14 21:18:19,769 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 21:18:19,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:19,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:18:19,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:19,773 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:18:19,781 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:18:19,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:19,783 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:18:19,783 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:18:28,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:28,964 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:18:28,972 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:18:28,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:28,978 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1491, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:18:28,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:28,980 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1491, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:18:52,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:18:52,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:18:52,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.05 seconds 2025-02-14 21:18:52,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:52,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23358.23 MB 2025-02-14 21:18:52,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28634.80 MB 2025-02-14 21:18:52,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5276.57 MB 2025-02-14 21:18:52,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39950.75 MB 2025-02-14 21:18:52,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35949.38 MB 2025-02-14 21:18:52,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4001.37 MB 2025-02-14 21:18:52,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37586.75 MB 2025-02-14 21:18:52,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:18:52,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:18:52,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:18:52,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:52,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28634.80 MB 2025-02-14 21:18:52,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23529.10 MB 2025-02-14 21:18:52,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5105.70 MB 2025-02-14 21:18:52,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35949.38 MB 2025-02-14 21:18:52,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47523.56 MB 2025-02-14 21:18:52,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11574.18 MB 2025-02-14 21:18:52,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42664.18 MB 2025-02-14 21:18:54,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:18:54,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:18:54,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 21:18:54,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23529.10 MB 2025-02-14 21:18:54,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.94 MB 2025-02-14 21:18:54,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:18:54,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47523.56 MB 2025-02-14 21:18:54,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30670.85 MB 2025-02-14 21:18:54,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16852.71 MB 2025-02-14 21:18:54,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28038.48 MB 2025-02-14 21:18:54,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:18:54,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:18:54,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:18:54,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.94 MB 2025-02-14 21:18:54,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25949.47 MB 2025-02-14 21:18:54,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:18:54,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30670.85 MB 2025-02-14 21:18:54,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30670.85 MB 2025-02-14 21:18:54,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:54,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27366.90 MB 2025-02-14 21:18:54,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:18:54,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:18:54,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:18:54,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25949.47 MB 2025-02-14 21:18:54,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28191.33 MB 2025-02-14 21:18:54,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:18:54,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30670.85 MB 2025-02-14 21:18:54,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35861.30 MB 2025-02-14 21:18:54,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:18:54,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33735.61 MB 2025-02-14 21:18:54,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:18:54,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:18:54,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:18:54,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.94 MB 2025-02-14 21:18:54,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28191.33 MB 2025-02-14 21:18:54,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:18:54,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30670.85 MB 2025-02-14 21:18:54,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35861.30 MB 2025-02-14 21:18:54,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:18:54,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33735.61 MB 2025-02-14 21:18:54,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:18:54,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:18:54,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:18:54,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29724.87 MB 2025-02-14 21:18:54,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30491.87 MB 2025-02-14 21:18:54,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:18:54,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35861.30 MB 2025-02-14 21:18:54,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36278.63 MB 2025-02-14 21:18:54,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:18:54,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.66 MB 2025-02-14 21:18:54,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:18:54,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:18:54,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:18:54,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30904.76 MB 2025-02-14 21:18:54,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31132.30 MB 2025-02-14 21:18:54,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.54 MB 2025-02-14 21:18:54,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36278.63 MB 2025-02-14 21:18:54,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36278.63 MB 2025-02-14 21:18:54,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:54,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31338.87 MB 2025-02-14 21:18:54,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:18:54,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:18:54,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.47 seconds 2025-02-14 21:18:54,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18163.47 MB 2025-02-14 21:18:54,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31333.17 MB 2025-02-14 21:18:54,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13169.70 MB 2025-02-14 21:18:54,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39950.75 MB 2025-02-14 21:18:54,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36278.63 MB 2025-02-14 21:18:54,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3672.11 MB 2025-02-14 21:18:54,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31338.87 MB 2025-02-14 21:18:54,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:18:54,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:18:54,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:18:54,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31333.17 MB 2025-02-14 21:18:54,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23164.81 MB 2025-02-14 21:18:54,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8168.36 MB 2025-02-14 21:18:54,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36278.63 MB 2025-02-14 21:18:54,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36278.63 MB 2025-02-14 21:18:54,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:18:54,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33842.38 MB 2025-02-14 21:18:54,737 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 21:18:54,737 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:18:54,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:18:54,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:18:54,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:18:54,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:18:54,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23164.81 MB 2025-02-14 21:18:54,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31595.49 MB 2025-02-14 21:18:54,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 21:18:54,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36278.63 MB 2025-02-14 21:18:54,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44660.95 MB 2025-02-14 21:18:54,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 21:18:54,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31595.49 MB 2025-02-14 21:18:54,903 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 21:18:54,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:54,904 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:18:54,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:54,905 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:18:54,910 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:18:54,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:18:54,911 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:18:54,911 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:20:10,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:20:10,037 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:20:10,042 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:20:10,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:20:10,046 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 170, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:20:10,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:20:10,047 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 170, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:20:12,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:20:12,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:20:12,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.61 seconds 2025-02-14 21:20:12,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:12,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14153.59 MB 2025-02-14 21:20:12,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14755.21 MB 2025-02-14 21:20:12,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 601.62 MB 2025-02-14 21:20:12,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57233.38 MB 2025-02-14 21:20:12,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:20:12,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37702.60 MB 2025-02-14 21:20:12,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23624.96 MB 2025-02-14 21:20:12,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:20:12,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:20:12,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:20:12,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:12,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14755.21 MB 2025-02-14 21:20:12,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14983.49 MB 2025-02-14 21:20:12,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-14 21:20:12,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:20:12,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:20:12,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:20:12,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17052.08 MB 2025-02-14 21:20:13,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:20:13,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:20:13,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 21:20:13,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14983.49 MB 2025-02-14 21:20:13,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15197.15 MB 2025-02-14 21:20:13,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 21:20:13,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:20:13,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:20:13,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:20:13,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19154.17 MB 2025-02-14 21:20:13,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:20:13,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:20:13,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:20:13,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15197.08 MB 2025-02-14 21:20:13,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15957.44 MB 2025-02-14 21:20:13,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 21:20:13,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:20:13,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 21:20:13,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:20:13,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16527.96 MB 2025-02-14 21:20:13,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:20:13,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:20:13,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:20:13,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15957.44 MB 2025-02-14 21:20:13,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16859.82 MB 2025-02-14 21:20:13,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 21:20:13,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:20:13,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20675.82 MB 2025-02-14 21:20:13,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 21:20:13,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19091.36 MB 2025-02-14 21:20:13,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:20:13,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:20:13,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:20:13,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15197.08 MB 2025-02-14 21:20:13,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16859.82 MB 2025-02-14 21:20:13,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 21:20:13,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 21:20:13,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20675.82 MB 2025-02-14 21:20:13,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 21:20:13,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19091.36 MB 2025-02-14 21:20:13,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:20:13,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:20:13,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:20:13,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17477.07 MB 2025-02-14 21:20:13,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17785.79 MB 2025-02-14 21:20:13,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 21:20:13,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20675.82 MB 2025-02-14 21:20:13,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 21:20:13,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 21:20:13,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18077.69 MB 2025-02-14 21:20:13,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:20:13,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:20:13,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:20:13,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17951.99 MB 2025-02-14 21:20:13,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18180.07 MB 2025-02-14 21:20:13,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 21:20:13,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 21:20:13,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 21:20:13,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:20:13,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18194.75 MB 2025-02-14 21:20:13,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:20:13,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:20:13,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.59 seconds 2025-02-14 21:20:13,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13561.29 MB 2025-02-14 21:20:13,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18380.99 MB 2025-02-14 21:20:13,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4819.70 MB 2025-02-14 21:20:13,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57233.38 MB 2025-02-14 21:20:13,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 21:20:13,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36391.88 MB 2025-02-14 21:20:13,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18380.99 MB 2025-02-14 21:20:13,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:20:13,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:20:13,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:20:13,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18380.99 MB 2025-02-14 21:20:13,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17435.49 MB 2025-02-14 21:20:13,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -945.51 MB 2025-02-14 21:20:13,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 21:20:13,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 21:20:13,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:20:13,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19184.14 MB 2025-02-14 21:20:13,921 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 21:20:13,921 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:20:13,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:20:13,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:20:13,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:20:13,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:20:13,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17435.49 MB 2025-02-14 21:20:13,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25868.79 MB 2025-02-14 21:20:13,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 21:20:13,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 21:20:13,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29225.91 MB 2025-02-14 21:20:13,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 21:20:13,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25868.79 MB 2025-02-14 21:20:14,085 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 21:20:14,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:20:14,087 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:20:14,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:20:14,088 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:20:14,092 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:20:14,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:20:14,093 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:20:14,094 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:21:15,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:21:15,651 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:21:15,656 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:21:15,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:21:15,660 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1520, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:21:15,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:21:15,661 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1520, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:21:38,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:21:38,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:21:38,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.24 seconds 2025-02-14 21:21:38,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:38,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23560.31 MB 2025-02-14 21:21:38,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28939.50 MB 2025-02-14 21:21:38,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5379.19 MB 2025-02-14 21:21:38,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37610.32 MB 2025-02-14 21:21:38,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36071.01 MB 2025-02-14 21:21:38,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1539.31 MB 2025-02-14 21:21:38,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37788.83 MB 2025-02-14 21:21:38,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:21:38,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:21:38,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:21:38,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:38,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.50 MB 2025-02-14 21:21:38,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23679.86 MB 2025-02-14 21:21:38,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5259.65 MB 2025-02-14 21:21:38,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36071.01 MB 2025-02-14 21:21:38,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47219.47 MB 2025-02-14 21:21:38,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11148.46 MB 2025-02-14 21:21:38,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42560.93 MB 2025-02-14 21:21:40,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:21:40,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:21:40,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 21:21:40,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:40,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23679.86 MB 2025-02-14 21:21:40,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24210.70 MB 2025-02-14 21:21:40,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:21:40,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47219.47 MB 2025-02-14 21:21:40,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30691.82 MB 2025-02-14 21:21:40,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16527.65 MB 2025-02-14 21:21:40,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28189.25 MB 2025-02-14 21:21:40,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:21:40,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:21:40,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:21:40,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:40,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24210.70 MB 2025-02-14 21:21:40,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26100.23 MB 2025-02-14 21:21:40,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:21:40,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30691.82 MB 2025-02-14 21:21:40,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30691.82 MB 2025-02-14 21:21:40,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:21:40,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27517.66 MB 2025-02-14 21:21:41,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:21:41,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:21:41,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:21:41,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:41,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26100.23 MB 2025-02-14 21:21:41,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28342.09 MB 2025-02-14 21:21:41,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:21:41,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30691.82 MB 2025-02-14 21:21:41,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-14 21:21:41,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:21:41,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33886.37 MB 2025-02-14 21:21:41,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:21:41,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:21:41,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:21:41,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:41,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24210.70 MB 2025-02-14 21:21:41,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28342.09 MB 2025-02-14 21:21:41,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:21:41,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30691.82 MB 2025-02-14 21:21:41,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36354.13 MB 2025-02-14 21:21:41,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:21:41,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33886.37 MB 2025-02-14 21:21:41,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:21:41,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:21:41,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:21:41,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:41,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29875.63 MB 2025-02-14 21:21:41,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30642.63 MB 2025-02-14 21:21:41,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:21:41,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36354.13 MB 2025-02-14 21:21:41,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36769.37 MB 2025-02-14 21:21:41,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:21:41,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31350.42 MB 2025-02-14 21:21:41,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:21:41,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:21:41,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:21:41,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:41,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31055.52 MB 2025-02-14 21:21:41,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31283.79 MB 2025-02-14 21:21:41,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.27 MB 2025-02-14 21:21:41,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36769.37 MB 2025-02-14 21:21:41,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36769.37 MB 2025-02-14 21:21:41,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:21:41,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31491.16 MB 2025-02-14 21:21:41,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:21:41,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:21:41,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.65 seconds 2025-02-14 21:21:41,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:41,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18264.51 MB 2025-02-14 21:21:41,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31484.28 MB 2025-02-14 21:21:41,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13219.77 MB 2025-02-14 21:21:41,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37610.32 MB 2025-02-14 21:21:41,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36769.37 MB 2025-02-14 21:21:41,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -840.96 MB 2025-02-14 21:21:41,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31491.16 MB 2025-02-14 21:21:41,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:21:41,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:21:41,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:21:41,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:41,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31484.28 MB 2025-02-14 21:21:41,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23255.48 MB 2025-02-14 21:21:41,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8228.80 MB 2025-02-14 21:21:41,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36769.37 MB 2025-02-14 21:21:41,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36769.37 MB 2025-02-14 21:21:41,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:21:41,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33984.88 MB 2025-02-14 21:21:41,592 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-14 21:21:41,592 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:21:41,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:21:41,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:21:41,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:21:41,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:21:41,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23255.48 MB 2025-02-14 21:21:41,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31657.01 MB 2025-02-14 21:21:41,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-14 21:21:41,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36769.37 MB 2025-02-14 21:21:41,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45124.42 MB 2025-02-14 21:21:41,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 21:21:41,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31657.01 MB 2025-02-14 21:21:41,753 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-14 21:21:41,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:21:41,755 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:21:41,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:21:41,756 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:21:41,760 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:21:41,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:21:41,761 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:21:41,761 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:22:33,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:22:33,691 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:22:33,696 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:22:33,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:22:33,700 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1624, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:22:33,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:22:33,701 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1624, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:22:58,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:22:58,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:22:58,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.98 seconds 2025-02-14 21:22:58,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:22:58,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24285.00 MB 2025-02-14 21:22:58,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30033.29 MB 2025-02-14 21:22:58,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5748.29 MB 2025-02-14 21:22:58,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53479.47 MB 2025-02-14 21:22:58,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36398.17 MB 2025-02-14 21:22:58,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17081.30 MB 2025-02-14 21:22:58,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38966.50 MB 2025-02-14 21:22:58,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:22:58,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:22:58,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:22:58,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:22:58,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30033.29 MB 2025-02-14 21:22:58,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24220.52 MB 2025-02-14 21:22:58,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5812.77 MB 2025-02-14 21:22:58,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36398.17 MB 2025-02-14 21:22:58,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47089.45 MB 2025-02-14 21:22:58,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10691.28 MB 2025-02-14 21:22:58,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41331.40 MB 2025-02-14 21:23:00,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:23:00,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:23:00,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 21:23:00,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:00,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24220.52 MB 2025-02-14 21:23:00,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24751.36 MB 2025-02-14 21:23:00,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:23:00,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47089.45 MB 2025-02-14 21:23:00,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32065.45 MB 2025-02-14 21:23:00,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15024.00 MB 2025-02-14 21:23:00,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28729.91 MB 2025-02-14 21:23:00,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:23:00,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:23:00,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:23:00,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:00,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24751.36 MB 2025-02-14 21:23:00,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26640.90 MB 2025-02-14 21:23:00,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:23:00,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32065.45 MB 2025-02-14 21:23:00,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32065.45 MB 2025-02-14 21:23:00,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:00,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28058.32 MB 2025-02-14 21:23:00,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:23:00,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:23:00,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:23:00,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:00,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26640.90 MB 2025-02-14 21:23:00,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28882.75 MB 2025-02-14 21:23:00,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:23:00,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32065.45 MB 2025-02-14 21:23:00,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36784.05 MB 2025-02-14 21:23:00,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:23:00,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34427.03 MB 2025-02-14 21:23:00,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:23:00,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:23:00,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:23:00,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:00,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24751.36 MB 2025-02-14 21:23:00,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28882.75 MB 2025-02-14 21:23:00,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:23:00,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32065.45 MB 2025-02-14 21:23:00,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36784.05 MB 2025-02-14 21:23:00,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:23:00,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34427.03 MB 2025-02-14 21:23:01,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:23:01,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:23:01,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:23:01,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:01,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30416.29 MB 2025-02-14 21:23:01,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31183.30 MB 2025-02-14 21:23:01,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:23:01,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36784.05 MB 2025-02-14 21:23:01,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37197.19 MB 2025-02-14 21:23:01,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 21:23:01,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31891.08 MB 2025-02-14 21:23:01,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:23:01,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:23:01,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:23:01,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:01,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31596.19 MB 2025-02-14 21:23:01,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31823.84 MB 2025-02-14 21:23:01,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.66 MB 2025-02-14 21:23:01,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37197.19 MB 2025-02-14 21:23:01,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37197.19 MB 2025-02-14 21:23:01,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:01,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32038.64 MB 2025-02-14 21:23:01,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:23:01,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:23:01,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.39 seconds 2025-02-14 21:23:01,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:01,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18626.85 MB 2025-02-14 21:23:01,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32024.92 MB 2025-02-14 21:23:01,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13398.06 MB 2025-02-14 21:23:01,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53479.47 MB 2025-02-14 21:23:01,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37197.19 MB 2025-02-14 21:23:01,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16282.29 MB 2025-02-14 21:23:01,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32038.64 MB 2025-02-14 21:23:01,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:23:01,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:23:01,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:23:01,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:01,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32024.92 MB 2025-02-14 21:23:01,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23631.24 MB 2025-02-14 21:23:01,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8393.67 MB 2025-02-14 21:23:01,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37197.19 MB 2025-02-14 21:23:01,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37197.19 MB 2025-02-14 21:23:01,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:01,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34536.58 MB 2025-02-14 21:23:01,379 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:23:01,379 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:23:01,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:23:01,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:23:01,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:23:01,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:01,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23631.24 MB 2025-02-14 21:23:01,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32070.26 MB 2025-02-14 21:23:01,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:23:01,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37197.19 MB 2025-02-14 21:23:01,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45587.89 MB 2025-02-14 21:23:01,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:23:01,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32070.26 MB 2025-02-14 21:23:01,541 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:23:01,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:01,543 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:23:01,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:01,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:23:01,548 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:23:01,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:01,549 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:23:01,550 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:23:07,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:07,712 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:23:07,717 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:23:07,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:07,721 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:23:07,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:07,722 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:23:26,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:23:26,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:23:26,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.36 seconds 2025-02-14 21:23:26,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:26,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-14 21:23:26,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-14 21:23:26,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-14 21:23:26,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58172.90 MB 2025-02-14 21:23:26,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 21:23:26,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27512.54 MB 2025-02-14 21:23:26,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34308.28 MB 2025-02-14 21:23:26,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:23:26,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:23:26,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:23:26,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:26,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-14 21:23:26,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21927.90 MB 2025-02-14 21:23:26,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3470.71 MB 2025-02-14 21:23:26,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 21:23:26,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43757.08 MB 2025-02-14 21:23:26,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13096.71 MB 2025-02-14 21:23:26,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37984.72 MB 2025-02-14 21:23:28,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:23:28,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:23:28,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:23:28,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21927.90 MB 2025-02-14 21:23:28,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.74 MB 2025-02-14 21:23:28,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:23:28,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43757.08 MB 2025-02-14 21:23:28,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27887.93 MB 2025-02-14 21:23:28,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15869.15 MB 2025-02-14 21:23:28,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26437.29 MB 2025-02-14 21:23:28,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:23:28,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:23:28,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:23:28,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 21:23:28,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24348.27 MB 2025-02-14 21:23:28,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:23:28,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27887.93 MB 2025-02-14 21:23:28,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27887.93 MB 2025-02-14 21:23:28,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:28,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25765.70 MB 2025-02-14 21:23:28,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:23:28,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:23:28,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:23:28,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24348.27 MB 2025-02-14 21:23:28,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 21:23:28,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:23:28,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27887.93 MB 2025-02-14 21:23:28,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34022.10 MB 2025-02-14 21:23:28,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:23:28,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 21:23:28,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:23:28,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:23:28,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:23:28,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 21:23:28,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 21:23:28,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:23:28,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27887.93 MB 2025-02-14 21:23:28,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34022.10 MB 2025-02-14 21:23:28,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:23:28,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 21:23:28,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:23:28,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:23:28,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:23:28,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28123.67 MB 2025-02-14 21:23:28,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28890.67 MB 2025-02-14 21:23:28,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:23:28,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34022.10 MB 2025-02-14 21:23:28,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34435.24 MB 2025-02-14 21:23:28,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 21:23:28,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.46 MB 2025-02-14 21:23:28,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:23:28,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:23:28,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:23:28,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29303.56 MB 2025-02-14 21:23:28,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29531.74 MB 2025-02-14 21:23:28,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 21:23:28,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34435.24 MB 2025-02-14 21:23:28,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34435.24 MB 2025-02-14 21:23:28,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:28,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29761.70 MB 2025-02-14 21:23:28,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:23:28,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:23:28,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.78 seconds 2025-02-14 21:23:28,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-14 21:23:28,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29732.22 MB 2025-02-14 21:23:28,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12641.85 MB 2025-02-14 21:23:28,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58172.90 MB 2025-02-14 21:23:28,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34435.24 MB 2025-02-14 21:23:28,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23737.66 MB 2025-02-14 21:23:28,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29761.70 MB 2025-02-14 21:23:28,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:23:28,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:23:28,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:23:28,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29732.22 MB 2025-02-14 21:23:28,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22079.92 MB 2025-02-14 21:23:28,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7652.30 MB 2025-02-14 21:23:28,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34435.24 MB 2025-02-14 21:23:28,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34435.24 MB 2025-02-14 21:23:28,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:28,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32231.60 MB 2025-02-14 21:23:28,797 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 21:23:28,797 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 21:23:28,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:23:28,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:23:28,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:23:28,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:28,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22079.92 MB 2025-02-14 21:23:28,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30477.32 MB 2025-02-14 21:23:28,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-14 21:23:28,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34435.24 MB 2025-02-14 21:23:28,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42786.10 MB 2025-02-14 21:23:28,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 21:23:28,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30477.32 MB 2025-02-14 21:23:28,959 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 21:23:28,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:28,960 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:23:28,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:28,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:23:28,967 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:23:28,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:28,968 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:23:28,969 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 21:23:42,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:42,715 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:23:42,720 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:23:42,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:42,723 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 86, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:23:42,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:42,724 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 86, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:23:44,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:23:44,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:23:44,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.42 seconds 2025-02-14 21:23:44,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-14 21:23:44,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13872.32 MB 2025-02-14 21:23:44,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.35 MB 2025-02-14 21:23:44,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51136.95 MB 2025-02-14 21:23:44,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 21:23:44,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32549.90 MB 2025-02-14 21:23:44,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22812.85 MB 2025-02-14 21:23:44,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:23:44,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:23:44,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:23:44,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13872.32 MB 2025-02-14 21:23:44,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14019.77 MB 2025-02-14 21:23:44,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.46 MB 2025-02-14 21:23:44,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 21:23:44,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 21:23:44,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:44,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14476.36 MB 2025-02-14 21:23:44,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:23:44,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:23:44,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.43 seconds 2025-02-14 21:23:44,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14019.77 MB 2025-02-14 21:23:44,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14133.90 MB 2025-02-14 21:23:44,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 114.13 MB 2025-02-14 21:23:44,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 21:23:44,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 21:23:44,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:44,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18104.49 MB 2025-02-14 21:23:44,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:23:44,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:23:44,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:23:44,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 21:23:44,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14539.99 MB 2025-02-14 21:23:44,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.15 MB 2025-02-14 21:23:44,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 21:23:44,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 21:23:44,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:44,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.74 MB 2025-02-14 21:23:44,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:23:44,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:23:44,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:23:44,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14539.99 MB 2025-02-14 21:23:44,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 21:23:44,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 493.32 MB 2025-02-14 21:23:44,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 21:23:44,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 21:23:44,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:44,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-14 21:23:44,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:23:44,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:23:44,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:23:44,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 21:23:44,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 21:23:44,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 899.47 MB 2025-02-14 21:23:44,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 21:23:44,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 21:23:44,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:44,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-14 21:23:44,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:23:44,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:23:44,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:23:44,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15509.56 MB 2025-02-14 21:23:44,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15716.74 MB 2025-02-14 21:23:44,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.18 MB 2025-02-14 21:23:44,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 21:23:44,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18717.08 MB 2025-02-14 21:23:44,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 130.02 MB 2025-02-14 21:23:44,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15868.91 MB 2025-02-14 21:23:44,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:23:44,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:23:44,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:23:44,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15847.79 MB 2025-02-14 21:23:44,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16052.76 MB 2025-02-14 21:23:44,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.97 MB 2025-02-14 21:23:44,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18717.08 MB 2025-02-14 21:23:44,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18717.08 MB 2025-02-14 21:23:44,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:44,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16052.76 MB 2025-02-14 21:23:44,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:23:44,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:23:44,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.06 seconds 2025-02-14 21:23:44,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:44,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13268.34 MB 2025-02-14 21:23:44,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16236.25 MB 2025-02-14 21:23:44,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2967.91 MB 2025-02-14 21:23:44,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51136.95 MB 2025-02-14 21:23:44,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18717.08 MB 2025-02-14 21:23:44,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32419.87 MB 2025-02-14 21:23:44,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16236.25 MB 2025-02-14 21:23:45,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:23:45,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:23:45,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:23:45,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:45,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13767.57 MB 2025-02-14 21:23:45,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16518.02 MB 2025-02-14 21:23:45,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2750.45 MB 2025-02-14 21:23:45,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18717.08 MB 2025-02-14 21:23:45,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18717.08 MB 2025-02-14 21:23:45,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:23:45,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16793.03 MB 2025-02-14 21:23:45,071 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7447, cut from 7449 2025-02-14 21:23:45,071 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:23:45,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:23:45,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:23:45,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:23:45,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:23:45,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16518.02 MB 2025-02-14 21:23:45,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24218.84 MB 2025-02-14 21:23:45,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7700.82 MB 2025-02-14 21:23:45,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18717.08 MB 2025-02-14 21:23:45,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28288.48 MB 2025-02-14 21:23:45,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9571.40 MB 2025-02-14 21:23:45,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24218.84 MB 2025-02-14 21:23:45,303 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7239] 2025-02-14 21:23:45,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:45,306 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:23:45,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:45,308 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:23:45,315 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:23:45,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:23:45,317 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:23:45,317 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:24:37,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:37,705 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:24:37,710 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:24:37,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:37,714 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:24:37,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:37,715 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:24:42,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:24:42,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:24:42,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.46 seconds 2025-02-14 21:24:42,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:42,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14989.47 MB 2025-02-14 21:24:42,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16015.77 MB 2025-02-14 21:24:42,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1026.29 MB 2025-02-14 21:24:42,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39772.49 MB 2025-02-14 21:24:42,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27130.86 MB 2025-02-14 21:24:42,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12641.63 MB 2025-02-14 21:24:42,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24913.83 MB 2025-02-14 21:24:42,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:24:42,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:24:42,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:24:42,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:42,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16015.77 MB 2025-02-14 21:24:42,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16400.57 MB 2025-02-14 21:24:42,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 384.80 MB 2025-02-14 21:24:42,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27130.86 MB 2025-02-14 21:24:42,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27130.86 MB 2025-02-14 21:24:42,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:24:42,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19871.46 MB 2025-02-14 21:24:43,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:24:43,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:24:43,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.30 seconds 2025-02-14 21:24:43,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:43,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16400.57 MB 2025-02-14 21:24:43,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16764.20 MB 2025-02-14 21:24:43,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 363.63 MB 2025-02-14 21:24:43,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27130.86 MB 2025-02-14 21:24:43,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27130.86 MB 2025-02-14 21:24:43,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:24:43,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20740.09 MB 2025-02-14 21:24:43,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:24:43,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:24:43,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:24:43,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:43,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16764.20 MB 2025-02-14 21:24:43,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18058.23 MB 2025-02-14 21:24:43,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1294.03 MB 2025-02-14 21:24:43,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27130.86 MB 2025-02-14 21:24:43,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27130.86 MB 2025-02-14 21:24:43,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:24:43,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19029.17 MB 2025-02-14 21:24:43,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:24:43,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:24:43,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 21:24:43,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:43,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18058.23 MB 2025-02-14 21:24:43,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19593.92 MB 2025-02-14 21:24:43,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1535.69 MB 2025-02-14 21:24:43,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27130.86 MB 2025-02-14 21:24:43,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27130.86 MB 2025-02-14 21:24:43,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:24:43,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23391.73 MB 2025-02-14 21:24:43,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:24:43,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:24:43,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 21:24:43,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:43,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16764.20 MB 2025-02-14 21:24:43,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19593.92 MB 2025-02-14 21:24:43,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2829.73 MB 2025-02-14 21:24:43,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27130.86 MB 2025-02-14 21:24:43,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27130.86 MB 2025-02-14 21:24:43,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:24:43,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23391.73 MB 2025-02-14 21:24:43,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:24:43,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:24:43,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:24:43,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:43,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20644.40 MB 2025-02-14 21:24:43,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21169.79 MB 2025-02-14 21:24:43,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 525.40 MB 2025-02-14 21:24:43,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27130.86 MB 2025-02-14 21:24:43,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27416.07 MB 2025-02-14 21:24:43,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 285.21 MB 2025-02-14 21:24:43,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21654.63 MB 2025-02-14 21:24:43,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:24:43,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:24:43,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:24:43,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:43,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21452.63 MB 2025-02-14 21:24:43,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21658.80 MB 2025-02-14 21:24:43,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.17 MB 2025-02-14 21:24:43,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27416.07 MB 2025-02-14 21:24:43,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27420.26 MB 2025-02-14 21:24:43,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 21:24:43,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21757.43 MB 2025-02-14 21:24:43,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:24:43,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:24:43,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.08 seconds 2025-02-14 21:24:43,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:43,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-14 21:24:43,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21859.87 MB 2025-02-14 21:24:43,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7880.78 MB 2025-02-14 21:24:43,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39772.49 MB 2025-02-14 21:24:43,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27420.26 MB 2025-02-14 21:24:43,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12352.23 MB 2025-02-14 21:24:43,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21859.87 MB 2025-02-14 21:24:44,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:24:44,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:24:44,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:24:44,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:44,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21859.87 MB 2025-02-14 21:24:44,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24873.91 MB 2025-02-14 21:24:44,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 21:24:44,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27420.26 MB 2025-02-14 21:24:44,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27420.26 MB 2025-02-14 21:24:44,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:24:44,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25175.27 MB 2025-02-14 21:24:44,076 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:24:44,077 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:24:44,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:24:44,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:24:44,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:24:44,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:24:44,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18388.85 MB 2025-02-14 21:24:44,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26827.87 MB 2025-02-14 21:24:44,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:24:44,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27420.26 MB 2025-02-14 21:24:44,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35810.97 MB 2025-02-14 21:24:44,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:24:44,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26827.87 MB 2025-02-14 21:24:44,245 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:24:44,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:44,246 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:24:44,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:44,247 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:24:44,252 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:24:44,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:44,253 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:24:44,253 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:24:54,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:54,206 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:24:54,211 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:24:54,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:54,215 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1221, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:24:54,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:24:54,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1221, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:25:13,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:25:13,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:25:13,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.88 seconds 2025-02-14 21:25:13,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:13,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21476.83 MB 2025-02-14 21:25:13,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25797.88 MB 2025-02-14 21:25:13,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4321.05 MB 2025-02-14 21:25:13,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48395.98 MB 2025-02-14 21:25:13,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33732.69 MB 2025-02-14 21:25:13,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14663.29 MB 2025-02-14 21:25:13,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34798.57 MB 2025-02-14 21:25:13,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:25:13,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:25:13,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:25:13,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:13,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25797.88 MB 2025-02-14 21:25:13,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22125.45 MB 2025-02-14 21:25:13,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3672.43 MB 2025-02-14 21:25:13,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33732.69 MB 2025-02-14 21:25:13,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43205.53 MB 2025-02-14 21:25:13,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9472.84 MB 2025-02-14 21:25:13,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38671.68 MB 2025-02-14 21:25:15,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:25:15,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:25:15,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:25:15,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22125.45 MB 2025-02-14 21:25:15,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22656.29 MB 2025-02-14 21:25:15,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:25:15,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43205.53 MB 2025-02-14 21:25:15,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29410.46 MB 2025-02-14 21:25:15,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13795.07 MB 2025-02-14 21:25:15,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26634.84 MB 2025-02-14 21:25:15,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:25:15,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:25:15,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:25:15,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22656.29 MB 2025-02-14 21:25:15,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24545.82 MB 2025-02-14 21:25:15,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:25:15,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29410.46 MB 2025-02-14 21:25:15,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29410.46 MB 2025-02-14 21:25:15,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:15,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25963.25 MB 2025-02-14 21:25:15,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:25:15,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:25:15,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:25:15,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24545.82 MB 2025-02-14 21:25:15,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26787.68 MB 2025-02-14 21:25:15,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:25:15,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29410.46 MB 2025-02-14 21:25:15,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34600.91 MB 2025-02-14 21:25:15,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:25:15,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32331.96 MB 2025-02-14 21:25:15,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:25:15,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:25:15,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:25:15,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22656.29 MB 2025-02-14 21:25:15,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26787.68 MB 2025-02-14 21:25:15,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:25:15,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29410.46 MB 2025-02-14 21:25:15,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34600.91 MB 2025-02-14 21:25:15,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:25:15,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32331.96 MB 2025-02-14 21:25:15,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:25:15,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:25:15,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:25:15,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28321.22 MB 2025-02-14 21:25:15,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29088.22 MB 2025-02-14 21:25:15,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:25:15,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34600.91 MB 2025-02-14 21:25:15,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35018.24 MB 2025-02-14 21:25:15,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:25:15,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29796.01 MB 2025-02-14 21:25:15,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:25:15,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:25:15,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:25:15,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29501.11 MB 2025-02-14 21:25:15,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29730.27 MB 2025-02-14 21:25:15,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-14 21:25:15,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35018.24 MB 2025-02-14 21:25:15,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35018.24 MB 2025-02-14 21:25:15,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:15,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29956.96 MB 2025-02-14 21:25:15,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:25:15,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:25:15,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.29 seconds 2025-02-14 21:25:15,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17222.77 MB 2025-02-14 21:25:15,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29931.34 MB 2025-02-14 21:25:15,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12708.58 MB 2025-02-14 21:25:15,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48395.98 MB 2025-02-14 21:25:15,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35018.24 MB 2025-02-14 21:25:15,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13377.73 MB 2025-02-14 21:25:15,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29956.96 MB 2025-02-14 21:25:15,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:25:15,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:25:15,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:25:15,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29931.34 MB 2025-02-14 21:25:15,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22227.16 MB 2025-02-14 21:25:15,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7704.19 MB 2025-02-14 21:25:15,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35018.24 MB 2025-02-14 21:25:15,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35018.24 MB 2025-02-14 21:25:15,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:15,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32443.01 MB 2025-02-14 21:25:15,797 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:25:15,797 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:25:15,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:25:15,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:25:15,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:25:15,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:15,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22227.16 MB 2025-02-14 21:25:15,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30666.18 MB 2025-02-14 21:25:15,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:25:15,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35018.24 MB 2025-02-14 21:25:15,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43408.95 MB 2025-02-14 21:25:15,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:25:15,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30666.18 MB 2025-02-14 21:25:15,960 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:25:15,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:15,962 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:25:15,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:15,963 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:25:15,967 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:25:15,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:15,968 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:25:15,968 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:25:24,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:24,819 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:25:24,823 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:25:24,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:24,827 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:25:24,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:24,828 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:25:27,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:25:27,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:25:27,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.82 seconds 2025-02-14 21:25:27,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:27,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-14 21:25:27,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-14 21:25:27,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-14 21:25:27,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55993.96 MB 2025-02-14 21:25:27,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:25:27,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36465.28 MB 2025-02-14 21:25:27,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23701.31 MB 2025-02-14 21:25:27,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:25:27,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:25:27,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:25:27,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:27,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-14 21:25:27,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.51 MB 2025-02-14 21:25:27,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.02 MB 2025-02-14 21:25:27,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:25:27,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:25:27,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:27,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17234.73 MB 2025-02-14 21:25:28,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:25:28,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:25:28,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-14 21:25:28,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.51 MB 2025-02-14 21:25:28,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15304.14 MB 2025-02-14 21:25:28,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-14 21:25:28,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:25:28,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:25:28,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:28,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19252.16 MB 2025-02-14 21:25:28,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:25:28,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:25:28,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:25:28,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15304.07 MB 2025-02-14 21:25:28,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16092.76 MB 2025-02-14 21:25:28,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.69 MB 2025-02-14 21:25:28,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:25:28,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 21:25:28,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:28,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16684.55 MB 2025-02-14 21:25:28,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:25:28,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:25:28,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:25:28,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16092.76 MB 2025-02-14 21:25:28,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17028.78 MB 2025-02-14 21:25:28,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.01 MB 2025-02-14 21:25:28,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:25:28,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20317.21 MB 2025-02-14 21:25:28,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 788.53 MB 2025-02-14 21:25:28,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19344.00 MB 2025-02-14 21:25:28,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:25:28,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:25:28,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:25:28,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15304.07 MB 2025-02-14 21:25:28,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17028.78 MB 2025-02-14 21:25:28,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1724.70 MB 2025-02-14 21:25:28,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 21:25:28,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20317.21 MB 2025-02-14 21:25:28,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 788.53 MB 2025-02-14 21:25:28,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19344.00 MB 2025-02-14 21:25:28,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:25:28,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:25:28,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:25:28,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17669.03 MB 2025-02-14 21:25:28,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17989.25 MB 2025-02-14 21:25:28,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.22 MB 2025-02-14 21:25:28,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20317.21 MB 2025-02-14 21:25:28,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20491.27 MB 2025-02-14 21:25:28,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-14 21:25:28,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18292.61 MB 2025-02-14 21:25:28,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:25:28,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:25:28,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:25:28,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18161.64 MB 2025-02-14 21:25:28,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18388.37 MB 2025-02-14 21:25:28,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.73 MB 2025-02-14 21:25:28,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20491.27 MB 2025-02-14 21:25:28,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20491.27 MB 2025-02-14 21:25:28,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:28,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18409.93 MB 2025-02-14 21:25:28,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:25:28,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:25:28,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.83 seconds 2025-02-14 21:25:28,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-14 21:25:28,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18589.42 MB 2025-02-14 21:25:28,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4990.09 MB 2025-02-14 21:25:28,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55993.96 MB 2025-02-14 21:25:28,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20491.27 MB 2025-02-14 21:25:28,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35502.69 MB 2025-02-14 21:25:28,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18589.42 MB 2025-02-14 21:25:28,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:25:28,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:25:28,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:25:28,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18589.42 MB 2025-02-14 21:25:28,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17503.74 MB 2025-02-14 21:25:28,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1085.68 MB 2025-02-14 21:25:28,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20491.27 MB 2025-02-14 21:25:28,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20491.27 MB 2025-02-14 21:25:28,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:25:28,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19192.14 MB 2025-02-14 21:25:28,944 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 21:25:28,944 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:25:28,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:25:28,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:25:28,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:25:28,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:25:28,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17503.74 MB 2025-02-14 21:25:28,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25942.57 MB 2025-02-14 21:25:28,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 21:25:28,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20491.27 MB 2025-02-14 21:25:28,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30977.03 MB 2025-02-14 21:25:28,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 21:25:28,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.57 MB 2025-02-14 21:25:29,113 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 21:25:29,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:29,115 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:25:29,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:29,116 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:25:29,120 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:25:29,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:25:29,121 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:25:29,122 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:26:14,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:26:14,160 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:26:14,165 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:26:14,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:26:14,169 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:26:14,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:26:14,170 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:26:16,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:26:16,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:26:16,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.38 seconds 2025-02-14 21:26:16,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:16,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14041.80 MB 2025-02-14 21:26:16,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14586.80 MB 2025-02-14 21:26:16,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-14 21:26:16,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39365.64 MB 2025-02-14 21:26:16,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22724.74 MB 2025-02-14 21:26:16,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16640.90 MB 2025-02-14 21:26:16,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23513.17 MB 2025-02-14 21:26:16,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:26:16,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:26:16,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:26:16,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:16,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14586.80 MB 2025-02-14 21:26:16,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14850.85 MB 2025-02-14 21:26:16,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.05 MB 2025-02-14 21:26:16,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22724.74 MB 2025-02-14 21:26:16,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22724.74 MB 2025-02-14 21:26:16,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:26:16,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16785.35 MB 2025-02-14 21:26:17,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:26:17,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:26:17,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.74 seconds 2025-02-14 21:26:17,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14850.85 MB 2025-02-14 21:26:17,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15055.22 MB 2025-02-14 21:26:17,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.37 MB 2025-02-14 21:26:17,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22724.74 MB 2025-02-14 21:26:17,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22724.74 MB 2025-02-14 21:26:17,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:26:17,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19020.50 MB 2025-02-14 21:26:17,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:26:17,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:26:17,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:26:17,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15055.16 MB 2025-02-14 21:26:17,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15782.45 MB 2025-02-14 21:26:17,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 727.29 MB 2025-02-14 21:26:17,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22724.74 MB 2025-02-14 21:26:17,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22724.74 MB 2025-02-14 21:26:17,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:26:17,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16328.17 MB 2025-02-14 21:26:17,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:26:17,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:26:17,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:26:17,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15782.45 MB 2025-02-14 21:26:17,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16645.61 MB 2025-02-14 21:26:17,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 863.15 MB 2025-02-14 21:26:17,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22724.74 MB 2025-02-14 21:26:17,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22724.74 MB 2025-02-14 21:26:17,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:26:17,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18780.12 MB 2025-02-14 21:26:17,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:26:17,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:26:17,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:26:17,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15055.16 MB 2025-02-14 21:26:17,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16645.61 MB 2025-02-14 21:26:17,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1590.45 MB 2025-02-14 21:26:17,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22724.74 MB 2025-02-14 21:26:17,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22724.74 MB 2025-02-14 21:26:17,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:26:17,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18780.12 MB 2025-02-14 21:26:17,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:26:17,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:26:17,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:26:17,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17236.02 MB 2025-02-14 21:26:17,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17531.32 MB 2025-02-14 21:26:17,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 295.30 MB 2025-02-14 21:26:17,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22724.74 MB 2025-02-14 21:26:17,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22886.22 MB 2025-02-14 21:26:17,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-14 21:26:17,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17813.55 MB 2025-02-14 21:26:17,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:26:17,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:26:17,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:26:17,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17690.29 MB 2025-02-14 21:26:17,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17895.51 MB 2025-02-14 21:26:17,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.22 MB 2025-02-14 21:26:17,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22886.22 MB 2025-02-14 21:26:17,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22890.41 MB 2025-02-14 21:26:17,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 21:26:17,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17914.65 MB 2025-02-14 21:26:17,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:26:17,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:26:17,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.32 seconds 2025-02-14 21:26:17,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13505.25 MB 2025-02-14 21:26:17,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18096.58 MB 2025-02-14 21:26:17,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4591.33 MB 2025-02-14 21:26:17,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39365.64 MB 2025-02-14 21:26:17,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22890.41 MB 2025-02-14 21:26:17,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16475.23 MB 2025-02-14 21:26:17,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18096.58 MB 2025-02-14 21:26:17,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:26:17,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:26:17,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:26:17,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18096.58 MB 2025-02-14 21:26:17,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17348.70 MB 2025-02-14 21:26:17,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -747.88 MB 2025-02-14 21:26:17,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22890.41 MB 2025-02-14 21:26:17,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22890.41 MB 2025-02-14 21:26:17,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:26:17,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19000.78 MB 2025-02-14 21:26:17,771 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:26:17,771 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:26:17,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:26:17,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:26:17,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:26:17,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:26:17,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17348.70 MB 2025-02-14 21:26:17,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25787.72 MB 2025-02-14 21:26:17,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:26:17,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22890.41 MB 2025-02-14 21:26:17,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31281.12 MB 2025-02-14 21:26:17,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:26:17,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25787.72 MB 2025-02-14 21:26:17,940 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:26:17,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:26:17,941 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:26:17,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:26:17,942 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:26:17,947 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:26:17,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:26:17,948 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:26:17,948 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:28:30,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:28:30,533 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:28:30,538 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:28:30,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:28:30,542 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1001, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:28:30,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:28:30,543 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1001, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:28:45,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:28:45,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:28:45,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.25 seconds 2025-02-14 21:28:45,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:45,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19943.83 MB 2025-02-14 21:28:45,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23486.32 MB 2025-02-14 21:28:45,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3542.48 MB 2025-02-14 21:28:45,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43866.13 MB 2025-02-14 21:28:45,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29125.25 MB 2025-02-14 21:28:45,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14740.88 MB 2025-02-14 21:28:45,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32360.41 MB 2025-02-14 21:28:45,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:28:45,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:28:45,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:28:45,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:45,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23486.32 MB 2025-02-14 21:28:45,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20981.74 MB 2025-02-14 21:28:45,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2504.58 MB 2025-02-14 21:28:45,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29125.25 MB 2025-02-14 21:28:45,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38054.92 MB 2025-02-14 21:28:45,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8929.67 MB 2025-02-14 21:28:45,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34227.35 MB 2025-02-14 21:28:47,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:28:47,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:28:47,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 21:28:47,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:47,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20981.74 MB 2025-02-14 21:28:47,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21512.58 MB 2025-02-14 21:28:47,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:28:47,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38054.92 MB 2025-02-14 21:28:47,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26996.64 MB 2025-02-14 21:28:47,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11058.28 MB 2025-02-14 21:28:47,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25491.13 MB 2025-02-14 21:28:47,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:28:47,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:28:47,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:28:47,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:47,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21512.58 MB 2025-02-14 21:28:47,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23402.11 MB 2025-02-14 21:28:47,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:28:47,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26996.64 MB 2025-02-14 21:28:47,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27940.36 MB 2025-02-14 21:28:47,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 21:28:47,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24819.54 MB 2025-02-14 21:28:47,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:28:47,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:28:47,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:28:47,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:47,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23402.11 MB 2025-02-14 21:28:47,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25643.97 MB 2025-02-14 21:28:47,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:28:47,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27940.36 MB 2025-02-14 21:28:47,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33604.76 MB 2025-02-14 21:28:47,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 21:28:47,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31188.97 MB 2025-02-14 21:28:47,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:28:47,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:28:47,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:28:47,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:47,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21512.58 MB 2025-02-14 21:28:47,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25643.97 MB 2025-02-14 21:28:47,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:28:47,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26996.64 MB 2025-02-14 21:28:47,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33604.76 MB 2025-02-14 21:28:47,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-14 21:28:47,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31188.97 MB 2025-02-14 21:28:48,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:28:48,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:28:48,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 21:28:48,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:48,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27178.23 MB 2025-02-14 21:28:48,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27945.23 MB 2025-02-14 21:28:48,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:28:48,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33604.76 MB 2025-02-14 21:28:48,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34022.10 MB 2025-02-14 21:28:48,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:28:48,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28653.02 MB 2025-02-14 21:28:48,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:28:48,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:28:48,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:28:48,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:48,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28358.12 MB 2025-02-14 21:28:48,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28584.44 MB 2025-02-14 21:28:48,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.32 MB 2025-02-14 21:28:48,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34022.10 MB 2025-02-14 21:28:48,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34022.10 MB 2025-02-14 21:28:48,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:28:48,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28794.63 MB 2025-02-14 21:28:48,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:28:48,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:28:48,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.63 seconds 2025-02-14 21:28:48,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:48,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16456.27 MB 2025-02-14 21:28:48,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28784.48 MB 2025-02-14 21:28:48,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12328.21 MB 2025-02-14 21:28:48,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43866.13 MB 2025-02-14 21:28:48,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34022.10 MB 2025-02-14 21:28:48,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9844.03 MB 2025-02-14 21:28:48,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28794.63 MB 2025-02-14 21:28:48,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:28:48,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:28:48,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:28:48,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:48,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28784.48 MB 2025-02-14 21:28:48,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21445.82 MB 2025-02-14 21:28:48,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7338.66 MB 2025-02-14 21:28:48,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34022.10 MB 2025-02-14 21:28:48,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34022.10 MB 2025-02-14 21:28:48,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:28:48,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31283.25 MB 2025-02-14 21:28:48,463 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 21:28:48,463 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:28:48,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:28:48,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:28:48,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:28:48,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:28:48,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21445.82 MB 2025-02-14 21:28:48,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29842.47 MB 2025-02-14 21:28:48,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-14 21:28:48,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34022.10 MB 2025-02-14 21:28:48,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42368.76 MB 2025-02-14 21:28:48,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 21:28:48,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29842.47 MB 2025-02-14 21:28:48,632 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 21:28:48,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:28:48,633 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:28:48,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:28:48,634 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:28:48,639 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:28:48,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:28:48,640 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:28:48,640 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:29:45,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:29:45,235 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:29:45,240 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:29:45,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:29:45,243 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2934, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:29:45,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:29:45,244 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2934, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:30:30,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:30:30,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:30:30,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.02 seconds 2025-02-14 21:30:30,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:30,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33415.94 MB 2025-02-14 21:30:30,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43799.20 MB 2025-02-14 21:30:30,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10383.26 MB 2025-02-14 21:30:30,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71162.66 MB 2025-02-14 21:30:30,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48945.43 MB 2025-02-14 21:30:30,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22217.23 MB 2025-02-14 21:30:30,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54182.46 MB 2025-02-14 21:30:30,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:30:30,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:30:30,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:30:30,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:30,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43799.20 MB 2025-02-14 21:30:30,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31032.53 MB 2025-02-14 21:30:30,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12766.67 MB 2025-02-14 21:30:30,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48945.43 MB 2025-02-14 21:30:30,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 85289.07 MB 2025-02-14 21:30:30,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 36343.64 MB 2025-02-14 21:30:30,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73534.86 MB 2025-02-14 21:30:32,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:30:32,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:30:32,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 21:30:32,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:32,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31032.53 MB 2025-02-14 21:30:32,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31563.38 MB 2025-02-14 21:30:32,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:30:32,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 85289.07 MB 2025-02-14 21:30:32,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34783.36 MB 2025-02-14 21:30:32,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -50505.71 MB 2025-02-14 21:30:32,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35542.96 MB 2025-02-14 21:30:32,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:30:32,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:30:32,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:30:32,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:32,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31563.38 MB 2025-02-14 21:30:32,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33452.91 MB 2025-02-14 21:30:32,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:30:32,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34783.36 MB 2025-02-14 21:30:32,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36670.80 MB 2025-02-14 21:30:32,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:30:32,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34870.34 MB 2025-02-14 21:30:32,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:30:32,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:30:32,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:30:32,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:32,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33452.91 MB 2025-02-14 21:30:32,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35694.77 MB 2025-02-14 21:30:32,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:30:32,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36670.80 MB 2025-02-14 21:30:32,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42804.97 MB 2025-02-14 21:30:32,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:30:32,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41239.05 MB 2025-02-14 21:30:32,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:30:32,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:30:32,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 21:30:32,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:32,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31563.38 MB 2025-02-14 21:30:32,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35694.77 MB 2025-02-14 21:30:32,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:30:32,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34783.36 MB 2025-02-14 21:30:32,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42804.97 MB 2025-02-14 21:30:32,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 21:30:32,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41239.05 MB 2025-02-14 21:30:32,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:30:32,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:30:32,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:30:32,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:32,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37228.31 MB 2025-02-14 21:30:32,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37995.31 MB 2025-02-14 21:30:32,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:30:32,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42804.97 MB 2025-02-14 21:30:32,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43222.30 MB 2025-02-14 21:30:32,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:30:32,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38703.10 MB 2025-02-14 21:30:32,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:30:32,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:30:32,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:30:32,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:32,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38408.20 MB 2025-02-14 21:30:32,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38636.67 MB 2025-02-14 21:30:32,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 21:30:32,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43222.30 MB 2025-02-14 21:30:32,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43222.30 MB 2025-02-14 21:30:32,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:30:32,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38863.91 MB 2025-02-14 21:30:32,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:30:32,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:30:32,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.62 seconds 2025-02-14 21:30:32,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:32,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23192.32 MB 2025-02-14 21:30:32,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38837.05 MB 2025-02-14 21:30:32,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15644.73 MB 2025-02-14 21:30:32,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60939.04 MB 2025-02-14 21:30:32,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43222.30 MB 2025-02-14 21:30:32,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17716.74 MB 2025-02-14 21:30:32,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38863.91 MB 2025-02-14 21:30:33,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:30:33,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:30:33,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:30:33,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:33,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38837.05 MB 2025-02-14 21:30:33,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28186.14 MB 2025-02-14 21:30:33,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10650.91 MB 2025-02-14 21:30:33,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43222.30 MB 2025-02-14 21:30:33,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43222.30 MB 2025-02-14 21:30:33,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:30:33,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41340.22 MB 2025-02-14 21:30:33,149 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 21:30:33,149 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:30:33,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:30:33,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:30:33,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:30:33,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:30:33,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28186.14 MB 2025-02-14 21:30:33,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36595.95 MB 2025-02-14 21:30:33,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 21:30:33,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43222.30 MB 2025-02-14 21:30:33,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47401.93 MB 2025-02-14 21:30:33,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 21:30:33,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36595.95 MB 2025-02-14 21:30:33,311 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 21:30:33,313 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:30:33,313 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:30:33,314 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:30:33,314 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:30:33,318 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:30:33,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:30:33,320 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:30:33,320 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:30:43,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:30:43,603 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:30:43,608 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:30:43,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:30:43,612 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1288, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:30:43,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:30:43,613 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1288, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:31:03,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:31:03,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:31:03,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.07 seconds 2025-02-14 21:31:03,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:03,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21943.70 MB 2025-02-14 21:31:03,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26502.90 MB 2025-02-14 21:31:03,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4559.21 MB 2025-02-14 21:31:03,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59942.90 MB 2025-02-14 21:31:03,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35209.08 MB 2025-02-14 21:31:03,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24733.81 MB 2025-02-14 21:31:03,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35491.93 MB 2025-02-14 21:31:03,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:31:03,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:31:03,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:31:03,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:03,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26502.90 MB 2025-02-14 21:31:03,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22473.76 MB 2025-02-14 21:31:03,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4029.14 MB 2025-02-14 21:31:03,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35209.08 MB 2025-02-14 21:31:03,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44835.01 MB 2025-02-14 21:31:03,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9625.93 MB 2025-02-14 21:31:03,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40136.65 MB 2025-02-14 21:31:05,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:31:05,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:31:05,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 21:31:05,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:05,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22473.76 MB 2025-02-14 21:31:05,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23004.60 MB 2025-02-14 21:31:05,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:31:05,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44835.01 MB 2025-02-14 21:31:05,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30649.88 MB 2025-02-14 21:31:05,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14185.14 MB 2025-02-14 21:31:05,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26983.15 MB 2025-02-14 21:31:05,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:31:05,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:31:05,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:31:05,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:05,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23004.60 MB 2025-02-14 21:31:05,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24894.14 MB 2025-02-14 21:31:05,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:31:05,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30649.88 MB 2025-02-14 21:31:05,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30651.97 MB 2025-02-14 21:31:05,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 21:31:05,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26311.57 MB 2025-02-14 21:31:05,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:31:05,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:31:05,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:31:05,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:05,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24894.14 MB 2025-02-14 21:31:05,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27135.99 MB 2025-02-14 21:31:05,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:31:05,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30651.97 MB 2025-02-14 21:31:05,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34898.71 MB 2025-02-14 21:31:05,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 21:31:05,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32680.27 MB 2025-02-14 21:31:05,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:31:05,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:31:05,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:31:05,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:05,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23004.60 MB 2025-02-14 21:31:05,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27135.99 MB 2025-02-14 21:31:05,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:31:05,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30649.88 MB 2025-02-14 21:31:05,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34898.71 MB 2025-02-14 21:31:05,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4248.83 MB 2025-02-14 21:31:05,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32680.27 MB 2025-02-14 21:31:06,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:31:06,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:31:06,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:31:06,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:06,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28669.53 MB 2025-02-14 21:31:06,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29436.54 MB 2025-02-14 21:31:06,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:31:06,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34898.71 MB 2025-02-14 21:31:06,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35316.04 MB 2025-02-14 21:31:06,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:31:06,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30144.33 MB 2025-02-14 21:31:06,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:31:06,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:31:06,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:31:06,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:06,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29849.43 MB 2025-02-14 21:31:06,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30077.53 MB 2025-02-14 21:31:06,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 21:31:06,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35316.04 MB 2025-02-14 21:31:06,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35316.04 MB 2025-02-14 21:31:06,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:06,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30316.64 MB 2025-02-14 21:31:06,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:31:06,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:31:06,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.49 seconds 2025-02-14 21:31:06,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:06,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17456.20 MB 2025-02-14 21:31:06,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30277.54 MB 2025-02-14 21:31:06,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12821.34 MB 2025-02-14 21:31:06,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59942.90 MB 2025-02-14 21:31:06,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35316.04 MB 2025-02-14 21:31:06,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24626.86 MB 2025-02-14 21:31:06,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30316.64 MB 2025-02-14 21:31:06,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:31:06,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:31:06,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:31:06,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:06,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30277.54 MB 2025-02-14 21:31:06,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22444.68 MB 2025-02-14 21:31:06,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7832.86 MB 2025-02-14 21:31:06,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35316.04 MB 2025-02-14 21:31:06,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35316.04 MB 2025-02-14 21:31:06,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:06,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32776.00 MB 2025-02-14 21:31:06,399 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 21:31:06,399 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:31:06,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:31:06,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:31:06,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:31:06,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:06,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22444.68 MB 2025-02-14 21:31:06,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30838.98 MB 2025-02-14 21:31:06,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8394.31 MB 2025-02-14 21:31:06,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35316.04 MB 2025-02-14 21:31:06,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39489.37 MB 2025-02-14 21:31:06,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 21:31:06,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30838.98 MB 2025-02-14 21:31:06,561 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 21:31:06,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:06,562 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:31:06,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:06,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:31:06,568 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:31:06,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:06,569 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:31:06,569 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:31:41,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:41,573 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:31:41,581 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:31:41,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:41,587 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:31:41,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:41,589 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:31:44,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:31:44,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:31:44,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.91 seconds 2025-02-14 21:31:44,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-14 21:31:44,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14881.00 MB 2025-02-14 21:31:44,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-14 21:31:44,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47836.04 MB 2025-02-14 21:31:44,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23238.54 MB 2025-02-14 21:31:44,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24597.50 MB 2025-02-14 21:31:44,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23708.28 MB 2025-02-14 21:31:44,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:31:44,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:31:44,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:31:44,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14881.00 MB 2025-02-14 21:31:44,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14350.30 MB 2025-02-14 21:31:44,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -530.70 MB 2025-02-14 21:31:44,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23238.54 MB 2025-02-14 21:31:44,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23238.54 MB 2025-02-14 21:31:44,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:44,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15773.15 MB 2025-02-14 21:31:44,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:31:44,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:31:44,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.31 seconds 2025-02-14 21:31:44,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14350.30 MB 2025-02-14 21:31:44,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14432.58 MB 2025-02-14 21:31:44,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 82.28 MB 2025-02-14 21:31:44,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23238.54 MB 2025-02-14 21:31:44,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23238.54 MB 2025-02-14 21:31:44,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:44,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18307.36 MB 2025-02-14 21:31:44,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:31:44,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:31:44,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:31:44,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14432.51 MB 2025-02-14 21:31:44,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14725.32 MB 2025-02-14 21:31:44,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.81 MB 2025-02-14 21:31:44,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23238.54 MB 2025-02-14 21:31:44,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23238.54 MB 2025-02-14 21:31:44,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:44,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14945.03 MB 2025-02-14 21:31:44,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:31:44,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:31:44,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:31:44,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14725.32 MB 2025-02-14 21:31:44,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15081.78 MB 2025-02-14 21:31:44,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.46 MB 2025-02-14 21:31:44,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23238.54 MB 2025-02-14 21:31:44,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23238.54 MB 2025-02-14 21:31:44,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:44,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15932.37 MB 2025-02-14 21:31:44,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:31:44,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:31:44,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:31:44,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14432.51 MB 2025-02-14 21:31:44,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15081.78 MB 2025-02-14 21:31:44,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 649.27 MB 2025-02-14 21:31:44,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23238.54 MB 2025-02-14 21:31:44,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23238.54 MB 2025-02-14 21:31:44,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:44,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15932.37 MB 2025-02-14 21:31:44,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:31:44,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:31:44,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 21:31:44,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15426.09 MB 2025-02-14 21:31:44,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15575.45 MB 2025-02-14 21:31:44,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 149.36 MB 2025-02-14 21:31:44,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23238.54 MB 2025-02-14 21:31:44,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23330.82 MB 2025-02-14 21:31:44,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 92.27 MB 2025-02-14 21:31:44,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15685.16 MB 2025-02-14 21:31:44,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:31:44,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:31:44,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:31:44,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15669.94 MB 2025-02-14 21:31:44,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15819.64 MB 2025-02-14 21:31:44,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 149.71 MB 2025-02-14 21:31:44,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23330.82 MB 2025-02-14 21:31:44,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23330.82 MB 2025-02-14 21:31:44,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:44,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15819.64 MB 2025-02-14 21:31:44,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:31:44,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:31:44,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-14 21:31:44,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:44,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13602.81 MB 2025-02-14 21:31:44,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15953.55 MB 2025-02-14 21:31:44,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2350.74 MB 2025-02-14 21:31:44,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47836.04 MB 2025-02-14 21:31:44,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23330.82 MB 2025-02-14 21:31:44,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24505.22 MB 2025-02-14 21:31:44,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15953.55 MB 2025-02-14 21:31:45,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:31:45,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:31:45,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 21:31:45,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:45,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15953.55 MB 2025-02-14 21:31:45,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15971.89 MB 2025-02-14 21:31:45,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 18.34 MB 2025-02-14 21:31:45,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23330.82 MB 2025-02-14 21:31:45,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23330.82 MB 2025-02-14 21:31:45,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:31:45,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17893.89 MB 2025-02-14 21:31:45,189 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5431, cut from 5433 2025-02-14 21:31:45,189 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:31:45,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:31:45,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:31:45,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:31:45,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:31:45,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15971.89 MB 2025-02-14 21:31:45,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21592.17 MB 2025-02-14 21:31:45,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5620.28 MB 2025-02-14 21:31:45,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23330.82 MB 2025-02-14 21:31:45,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26124.22 MB 2025-02-14 21:31:45,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2793.41 MB 2025-02-14 21:31:45,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21592.17 MB 2025-02-14 21:31:45,360 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5223] 2025-02-14 21:31:45,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:45,363 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:31:45,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:45,365 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:31:45,372 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:31:45,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:31:45,374 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:31:45,374 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:32:43,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:32:43,326 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:32:43,331 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:32:43,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:32:43,335 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 646, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:32:43,336 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:32:43,336 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 646, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:32:53,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:32:53,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:32:53,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.92 seconds 2025-02-14 21:32:53,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:53,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17470.14 MB 2025-02-14 21:32:53,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19756.30 MB 2025-02-14 21:32:53,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2286.16 MB 2025-02-14 21:32:53,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34506.54 MB 2025-02-14 21:32:53,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23668.46 MB 2025-02-14 21:32:53,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10838.08 MB 2025-02-14 21:32:53,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28753.45 MB 2025-02-14 21:32:53,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:32:53,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:32:53,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 21:32:53,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:53,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19756.30 MB 2025-02-14 21:32:53,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19137.25 MB 2025-02-14 21:32:53,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -619.04 MB 2025-02-14 21:32:53,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23668.46 MB 2025-02-14 21:32:53,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32262.59 MB 2025-02-14 21:32:53,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8594.13 MB 2025-02-14 21:32:53,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28393.42 MB 2025-02-14 21:32:55,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:32:55,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:32:55,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 21:32:55,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19137.25 MB 2025-02-14 21:32:55,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19668.09 MB 2025-02-14 21:32:55,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:32:55,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32262.59 MB 2025-02-14 21:32:55,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25792.87 MB 2025-02-14 21:32:55,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6469.71 MB 2025-02-14 21:32:55,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23646.64 MB 2025-02-14 21:32:55,217 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:32:55,217 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:32:55,217 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:32:55,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19668.09 MB 2025-02-14 21:32:55,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21557.63 MB 2025-02-14 21:32:55,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:32:55,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25792.87 MB 2025-02-14 21:32:55,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25794.97 MB 2025-02-14 21:32:55,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 21:32:55,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22975.06 MB 2025-02-14 21:32:55,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:32:55,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:32:55,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:32:55,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21557.63 MB 2025-02-14 21:32:55,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23799.48 MB 2025-02-14 21:32:55,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:32:55,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25794.97 MB 2025-02-14 21:32:55,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31457.28 MB 2025-02-14 21:32:55,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:32:55,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.77 MB 2025-02-14 21:32:55,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:32:55,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:32:55,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:32:55,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19668.09 MB 2025-02-14 21:32:55,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23799.48 MB 2025-02-14 21:32:55,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:32:55,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25792.87 MB 2025-02-14 21:32:55,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31457.28 MB 2025-02-14 21:32:55,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 21:32:55,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.77 MB 2025-02-14 21:32:55,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:32:55,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:32:55,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:32:55,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25333.03 MB 2025-02-14 21:32:55,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26100.03 MB 2025-02-14 21:32:55,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:32:55,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31457.28 MB 2025-02-14 21:32:55,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31874.61 MB 2025-02-14 21:32:55,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:32:55,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26807.82 MB 2025-02-14 21:32:55,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:32:55,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:32:55,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:32:55,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26512.92 MB 2025-02-14 21:32:55,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26741.71 MB 2025-02-14 21:32:55,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 21:32:55,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31874.61 MB 2025-02-14 21:32:55,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31874.61 MB 2025-02-14 21:32:55,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:32:55,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26936.02 MB 2025-02-14 21:32:55,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:32:55,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:32:55,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.27 seconds 2025-02-14 21:32:55,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.42 MB 2025-02-14 21:32:55,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26942.78 MB 2025-02-14 21:32:55,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11723.36 MB 2025-02-14 21:32:55,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34506.54 MB 2025-02-14 21:32:55,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31874.61 MB 2025-02-14 21:32:55,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2631.93 MB 2025-02-14 21:32:55,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26942.78 MB 2025-02-14 21:32:55,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:32:55,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:32:55,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:32:55,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26942.78 MB 2025-02-14 21:32:55,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20223.81 MB 2025-02-14 21:32:55,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6718.97 MB 2025-02-14 21:32:55,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31874.61 MB 2025-02-14 21:32:55,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31874.61 MB 2025-02-14 21:32:55,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:32:55,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29454.45 MB 2025-02-14 21:32:55,892 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:32:55,893 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:32:55,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:32:55,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:32:55,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:32:55,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:32:55,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20223.81 MB 2025-02-14 21:32:55,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28662.83 MB 2025-02-14 21:32:55,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:32:55,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31874.61 MB 2025-02-14 21:32:55,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40265.32 MB 2025-02-14 21:32:55,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:32:55,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28662.83 MB 2025-02-14 21:32:56,054 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:32:56,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:32:56,056 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:32:56,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:32:56,057 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:32:56,061 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:32:56,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:32:56,063 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:32:56,063 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:34:34,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:34:34,112 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:34:34,119 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:34:34,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:34:34,125 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:34:34,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:34:34,126 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:34:52,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:34:52,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:34:52,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.53 seconds 2025-02-14 21:34:52,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:52,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-14 21:34:52,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.80 MB 2025-02-14 21:34:52,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4278.58 MB 2025-02-14 21:34:52,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52850.33 MB 2025-02-14 21:34:52,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32145.15 MB 2025-02-14 21:34:52,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20705.18 MB 2025-02-14 21:34:52,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34488.46 MB 2025-02-14 21:34:52,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:34:52,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:34:52,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:34:52,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:52,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.80 MB 2025-02-14 21:34:52,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22064.11 MB 2025-02-14 21:34:52,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3607.68 MB 2025-02-14 21:34:52,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32145.15 MB 2025-02-14 21:34:52,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42853.20 MB 2025-02-14 21:34:52,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10708.06 MB 2025-02-14 21:34:52,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38020.29 MB 2025-02-14 21:34:54,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:34:54,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:34:54,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 21:34:54,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:54,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22064.11 MB 2025-02-14 21:34:54,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22594.96 MB 2025-02-14 21:34:54,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:34:54,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42853.20 MB 2025-02-14 21:34:54,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25809.65 MB 2025-02-14 21:34:54,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17043.55 MB 2025-02-14 21:34:54,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26574.54 MB 2025-02-14 21:34:54,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:34:54,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:34:54,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:34:54,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:54,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22594.96 MB 2025-02-14 21:34:54,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24484.49 MB 2025-02-14 21:34:54,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:34:54,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25809.65 MB 2025-02-14 21:34:54,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27697.09 MB 2025-02-14 21:34:54,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:34:54,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25901.92 MB 2025-02-14 21:34:54,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:34:54,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:34:54,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:34:54,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:54,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24484.49 MB 2025-02-14 21:34:54,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26726.35 MB 2025-02-14 21:34:54,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:34:54,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27697.09 MB 2025-02-14 21:34:54,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33831.26 MB 2025-02-14 21:34:54,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:34:54,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32270.63 MB 2025-02-14 21:34:54,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:34:54,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:34:54,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:34:54,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:54,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22594.96 MB 2025-02-14 21:34:54,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26726.35 MB 2025-02-14 21:34:54,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:34:54,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25809.65 MB 2025-02-14 21:34:54,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33831.26 MB 2025-02-14 21:34:54,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 21:34:54,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32270.63 MB 2025-02-14 21:34:55,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:34:55,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:34:55,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:34:55,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:55,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28259.89 MB 2025-02-14 21:34:55,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29026.89 MB 2025-02-14 21:34:55,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:34:55,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33831.26 MB 2025-02-14 21:34:55,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 21:34:55,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:34:55,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29734.68 MB 2025-02-14 21:34:55,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:34:55,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:34:55,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:34:55,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:55,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29439.78 MB 2025-02-14 21:34:55,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29667.47 MB 2025-02-14 21:34:55,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.69 MB 2025-02-14 21:34:55,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34248.59 MB 2025-02-14 21:34:55,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 21:34:55,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:34:55,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29880.90 MB 2025-02-14 21:34:55,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:34:55,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:34:55,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.92 seconds 2025-02-14 21:34:55,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:55,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17180.96 MB 2025-02-14 21:34:55,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29867.95 MB 2025-02-14 21:34:55,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12687.00 MB 2025-02-14 21:34:55,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52850.33 MB 2025-02-14 21:34:55,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 21:34:55,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18601.74 MB 2025-02-14 21:34:55,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29880.90 MB 2025-02-14 21:34:55,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:34:55,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:34:55,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:34:55,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:55,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29867.95 MB 2025-02-14 21:34:55,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22167.30 MB 2025-02-14 21:34:55,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7700.66 MB 2025-02-14 21:34:55,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34248.59 MB 2025-02-14 21:34:55,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 21:34:55,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:34:55,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32364.57 MB 2025-02-14 21:34:55,334 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 21:34:55,334 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:34:55,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:34:55,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:34:55,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:34:55,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:34:55,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22167.30 MB 2025-02-14 21:34:55,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30555.71 MB 2025-02-14 21:34:55,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 21:34:55,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34248.59 MB 2025-02-14 21:34:55,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42588.96 MB 2025-02-14 21:34:55,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-14 21:34:55,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30555.71 MB 2025-02-14 21:34:55,495 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 21:34:55,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:34:55,497 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:34:55,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:34:55,498 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:34:55,502 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:34:55,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:34:55,503 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:34:55,503 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:35:09,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:35:09,607 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:35:09,612 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:35:09,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:35:09,615 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2079, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:35:09,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:35:09,616 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2079, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:35:41,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:35:41,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:35:41,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.24 seconds 2025-02-14 21:35:41,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:41,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27455.51 MB 2025-02-14 21:35:41,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34812.98 MB 2025-02-14 21:35:41,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7357.46 MB 2025-02-14 21:35:41,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55098.47 MB 2025-02-14 21:35:41,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37981.52 MB 2025-02-14 21:35:41,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17116.95 MB 2025-02-14 21:35:41,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43722.46 MB 2025-02-14 21:35:42,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:35:42,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:35:42,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 21:35:42,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:42,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.98 MB 2025-02-14 21:35:42,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26586.97 MB 2025-02-14 21:35:42,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8226.00 MB 2025-02-14 21:35:42,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37981.52 MB 2025-02-14 21:35:42,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 68906.12 MB 2025-02-14 21:35:42,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30924.60 MB 2025-02-14 21:35:42,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56361.74 MB 2025-02-14 21:35:43,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:35:43,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:35:43,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 21:35:43,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:43,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26586.97 MB 2025-02-14 21:35:43,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27117.81 MB 2025-02-14 21:35:43,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:35:43,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68906.12 MB 2025-02-14 21:35:43,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33455.87 MB 2025-02-14 21:35:43,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35450.26 MB 2025-02-14 21:35:43,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31096.36 MB 2025-02-14 21:35:43,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:35:43,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:35:43,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:35:43,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:43,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-14 21:35:43,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29007.35 MB 2025-02-14 21:35:43,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:35:43,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33455.87 MB 2025-02-14 21:35:43,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33455.87 MB 2025-02-14 21:35:43,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:35:43,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30424.78 MB 2025-02-14 21:35:44,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:35:44,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:35:44,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:35:44,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:44,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29007.35 MB 2025-02-14 21:35:44,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-14 21:35:44,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:35:44,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33455.87 MB 2025-02-14 21:35:44,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38646.32 MB 2025-02-14 21:35:44,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:35:44,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-14 21:35:44,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:35:44,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:35:44,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:35:44,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:44,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.81 MB 2025-02-14 21:35:44,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31249.20 MB 2025-02-14 21:35:44,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:35:44,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33455.87 MB 2025-02-14 21:35:44,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38646.32 MB 2025-02-14 21:35:44,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:35:44,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.49 MB 2025-02-14 21:35:44,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:35:44,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:35:44,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:35:44,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:44,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32782.75 MB 2025-02-14 21:35:44,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33549.75 MB 2025-02-14 21:35:44,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:35:44,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38646.32 MB 2025-02-14 21:35:44,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39061.55 MB 2025-02-14 21:35:44,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:35:44,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34257.54 MB 2025-02-14 21:35:44,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:35:44,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:35:44,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:35:44,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:44,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33962.64 MB 2025-02-14 21:35:44,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34191.08 MB 2025-02-14 21:35:44,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 21:35:44,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39061.55 MB 2025-02-14 21:35:44,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39061.55 MB 2025-02-14 21:35:44,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:35:44,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34398.58 MB 2025-02-14 21:35:44,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:35:44,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:35:44,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.74 seconds 2025-02-14 21:35:44,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:44,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20212.11 MB 2025-02-14 21:35:44,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34391.44 MB 2025-02-14 21:35:44,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14179.33 MB 2025-02-14 21:35:44,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55098.47 MB 2025-02-14 21:35:44,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39061.55 MB 2025-02-14 21:35:44,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16036.92 MB 2025-02-14 21:35:44,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34398.58 MB 2025-02-14 21:35:44,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:35:44,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:35:44,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:35:44,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:44,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34391.44 MB 2025-02-14 21:35:44,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25205.57 MB 2025-02-14 21:35:44,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9185.87 MB 2025-02-14 21:35:44,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39061.55 MB 2025-02-14 21:35:44,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39061.55 MB 2025-02-14 21:35:44,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:35:44,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36894.32 MB 2025-02-14 21:35:44,650 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 21:35:44,650 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:35:44,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:35:44,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:35:44,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:35:44,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:35:44,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25205.57 MB 2025-02-14 21:35:44,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33614.88 MB 2025-02-14 21:35:44,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 21:35:44,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39061.55 MB 2025-02-14 21:35:44,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47420.80 MB 2025-02-14 21:35:44,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 21:35:44,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33614.88 MB 2025-02-14 21:35:44,813 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 21:35:44,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:35:44,814 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:35:44,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:35:44,815 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:35:44,819 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:35:44,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:35:44,821 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:35:44,821 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:36:35,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:36:35,156 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:36:35,161 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:36:35,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:36:35,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 225, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:36:35,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:36:35,166 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 225, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:36:38,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:36:38,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:36:38,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-14 21:36:38,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:38,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14536.54 MB 2025-02-14 21:36:38,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15332.80 MB 2025-02-14 21:36:38,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 796.26 MB 2025-02-14 21:36:38,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55780.05 MB 2025-02-14 21:36:38,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23465.03 MB 2025-02-14 21:36:38,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32315.02 MB 2025-02-14 21:36:38,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24234.41 MB 2025-02-14 21:36:38,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:36:38,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:36:38,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:36:38,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:38,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15332.80 MB 2025-02-14 21:36:38,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.26 MB 2025-02-14 21:36:38,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -21.55 MB 2025-02-14 21:36:38,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23465.03 MB 2025-02-14 21:36:38,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23465.03 MB 2025-02-14 21:36:38,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:36:38,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17682.10 MB 2025-02-14 21:36:39,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:36:39,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:36:39,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 21:36:39,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.26 MB 2025-02-14 21:36:39,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15532.88 MB 2025-02-14 21:36:39,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-14 21:36:39,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23465.03 MB 2025-02-14 21:36:39,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22993.17 MB 2025-02-14 21:36:39,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 21:36:39,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19480.91 MB 2025-02-14 21:36:39,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:36:39,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:36:39,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:36:39,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15532.82 MB 2025-02-14 21:36:39,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16321.51 MB 2025-02-14 21:36:39,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.69 MB 2025-02-14 21:36:39,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22993.17 MB 2025-02-14 21:36:39,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22993.17 MB 2025-02-14 21:36:39,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:36:39,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16913.29 MB 2025-02-14 21:36:39,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:36:39,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:36:39,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:36:39,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16321.51 MB 2025-02-14 21:36:39,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17257.52 MB 2025-02-14 21:36:39,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.01 MB 2025-02-14 21:36:39,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22993.17 MB 2025-02-14 21:36:39,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22993.17 MB 2025-02-14 21:36:39,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:36:39,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19572.22 MB 2025-02-14 21:36:39,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:36:39,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:36:39,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:36:39,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15532.82 MB 2025-02-14 21:36:39,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17257.52 MB 2025-02-14 21:36:39,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1724.70 MB 2025-02-14 21:36:39,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22993.17 MB 2025-02-14 21:36:39,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22993.17 MB 2025-02-14 21:36:39,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:36:39,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19572.22 MB 2025-02-14 21:36:39,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:36:39,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:36:39,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:36:39,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17897.77 MB 2025-02-14 21:36:39,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18218.00 MB 2025-02-14 21:36:39,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.22 MB 2025-02-14 21:36:39,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22993.17 MB 2025-02-14 21:36:39,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23165.14 MB 2025-02-14 21:36:39,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 171.97 MB 2025-02-14 21:36:39,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18521.40 MB 2025-02-14 21:36:39,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:36:39,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:36:39,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:36:39,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18390.38 MB 2025-02-14 21:36:39,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18617.27 MB 2025-02-14 21:36:39,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.89 MB 2025-02-14 21:36:39,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23165.14 MB 2025-02-14 21:36:39,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23165.14 MB 2025-02-14 21:36:39,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:36:39,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18642.20 MB 2025-02-14 21:36:39,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:36:39,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:36:39,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.45 seconds 2025-02-14 21:36:39,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13752.62 MB 2025-02-14 21:36:39,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18818.17 MB 2025-02-14 21:36:39,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5065.55 MB 2025-02-14 21:36:39,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55780.05 MB 2025-02-14 21:36:39,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23165.14 MB 2025-02-14 21:36:39,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32614.91 MB 2025-02-14 21:36:39,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18818.17 MB 2025-02-14 21:36:39,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:36:39,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:36:39,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:36:39,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18818.17 MB 2025-02-14 21:36:39,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17654.75 MB 2025-02-14 21:36:39,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1163.42 MB 2025-02-14 21:36:39,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23165.14 MB 2025-02-14 21:36:39,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23165.14 MB 2025-02-14 21:36:39,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:36:39,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19420.46 MB 2025-02-14 21:36:39,907 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 21:36:39,907 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:36:39,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:36:39,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:36:39,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:36:39,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:36:39,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17654.75 MB 2025-02-14 21:36:39,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26086.22 MB 2025-02-14 21:36:39,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 21:36:39,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23165.14 MB 2025-02-14 21:36:39,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31549.55 MB 2025-02-14 21:36:39,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 21:36:39,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26086.22 MB 2025-02-14 21:36:40,075 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 21:36:40,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:36:40,076 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:36:40,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:36:40,077 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:36:40,082 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:36:40,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:36:40,083 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:36:40,083 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:37:33,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:37:33,226 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:37:33,234 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:37:33,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:37:33,241 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:37:33,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:37:33,243 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:37:51,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:37:51,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:37:51,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.22 seconds 2025-02-14 21:37:51,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:51,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21232.94 MB 2025-02-14 21:37:51,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25430.13 MB 2025-02-14 21:37:51,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4197.19 MB 2025-02-14 21:37:51,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39933.97 MB 2025-02-14 21:37:51,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30672.95 MB 2025-02-14 21:37:51,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9261.02 MB 2025-02-14 21:37:51,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34328.19 MB 2025-02-14 21:37:51,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:37:51,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:37:51,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:37:51,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:51,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25430.13 MB 2025-02-14 21:37:51,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21943.49 MB 2025-02-14 21:37:51,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3486.64 MB 2025-02-14 21:37:51,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30672.95 MB 2025-02-14 21:37:51,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43757.08 MB 2025-02-14 21:37:51,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13084.13 MB 2025-02-14 21:37:51,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37957.21 MB 2025-02-14 21:37:53,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:37:53,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:37:53,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 21:37:53,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:53,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21943.49 MB 2025-02-14 21:37:53,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22474.34 MB 2025-02-14 21:37:53,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:37:53,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43757.08 MB 2025-02-14 21:37:53,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27890.02 MB 2025-02-14 21:37:53,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15867.05 MB 2025-02-14 21:37:53,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26452.88 MB 2025-02-14 21:37:53,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:37:53,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:37:53,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:37:53,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:53,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22474.34 MB 2025-02-14 21:37:53,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24363.87 MB 2025-02-14 21:37:53,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:37:53,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 21:37:53,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27890.02 MB 2025-02-14 21:37:53,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:37:53,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25781.30 MB 2025-02-14 21:37:53,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:37:53,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:37:53,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:37:53,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:53,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24363.87 MB 2025-02-14 21:37:53,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26605.73 MB 2025-02-14 21:37:53,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:37:53,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 21:37:53,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34024.19 MB 2025-02-14 21:37:53,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:37:53,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32150.01 MB 2025-02-14 21:37:53,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:37:53,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:37:53,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 21:37:53,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:53,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22474.34 MB 2025-02-14 21:37:53,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26605.73 MB 2025-02-14 21:37:53,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:37:53,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27890.02 MB 2025-02-14 21:37:53,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34024.19 MB 2025-02-14 21:37:53,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:37:53,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32150.01 MB 2025-02-14 21:37:53,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:37:53,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:37:53,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:37:53,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:53,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28139.27 MB 2025-02-14 21:37:53,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28906.27 MB 2025-02-14 21:37:53,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:37:53,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34024.19 MB 2025-02-14 21:37:53,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34441.53 MB 2025-02-14 21:37:53,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:37:53,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29614.06 MB 2025-02-14 21:37:53,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:37:53,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:37:53,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:37:53,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:53,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29319.16 MB 2025-02-14 21:37:53,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29547.08 MB 2025-02-14 21:37:53,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.92 MB 2025-02-14 21:37:53,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34441.53 MB 2025-02-14 21:37:53,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34441.53 MB 2025-02-14 21:37:53,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:37:53,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29781.98 MB 2025-02-14 21:37:53,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:37:53,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:37:53,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.65 seconds 2025-02-14 21:37:53,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:53,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17100.83 MB 2025-02-14 21:37:53,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29747.56 MB 2025-02-14 21:37:53,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12646.73 MB 2025-02-14 21:37:53,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39933.97 MB 2025-02-14 21:37:53,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34441.53 MB 2025-02-14 21:37:53,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5492.44 MB 2025-02-14 21:37:53,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29781.98 MB 2025-02-14 21:37:54,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:37:54,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:37:54,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:37:54,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:54,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29747.56 MB 2025-02-14 21:37:54,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22092.15 MB 2025-02-14 21:37:54,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7655.41 MB 2025-02-14 21:37:54,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34441.53 MB 2025-02-14 21:37:54,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34441.53 MB 2025-02-14 21:37:54,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:37:54,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32248.47 MB 2025-02-14 21:37:54,178 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 21:37:54,178 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:37:54,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:37:54,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:37:54,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:37:54,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:37:54,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22092.15 MB 2025-02-14 21:37:54,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30495.71 MB 2025-02-14 21:37:54,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 21:37:54,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34441.53 MB 2025-02-14 21:37:54,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42796.58 MB 2025-02-14 21:37:54,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 21:37:54,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30495.71 MB 2025-02-14 21:37:54,343 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 21:37:54,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:37:54,344 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:37:54,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:37:54,345 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:37:54,351 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:37:54,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:37:54,352 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:37:54,352 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:38:05,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:38:05,709 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:38:05,714 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:38:05,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:38:05,717 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1011, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:38:05,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:38:05,718 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1011, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:38:21,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:38:21,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:38:21,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.62 seconds 2025-02-14 21:38:21,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:21,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20013.52 MB 2025-02-14 21:38:21,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23591.39 MB 2025-02-14 21:38:21,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3577.87 MB 2025-02-14 21:38:21,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51151.63 MB 2025-02-14 21:38:21,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26474.45 MB 2025-02-14 21:38:21,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24677.19 MB 2025-02-14 21:38:21,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32429.29 MB 2025-02-14 21:38:21,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:38:21,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:38:21,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:38:21,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:21,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23591.39 MB 2025-02-14 21:38:21,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21034.77 MB 2025-02-14 21:38:21,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2556.62 MB 2025-02-14 21:38:21,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26474.45 MB 2025-02-14 21:38:21,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42020.63 MB 2025-02-14 21:38:21,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15546.19 MB 2025-02-14 21:38:21,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34711.19 MB 2025-02-14 21:38:23,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:38:23,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:38:23,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 21:38:23,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:23,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21034.77 MB 2025-02-14 21:38:23,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21565.62 MB 2025-02-14 21:38:23,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:38:23,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42020.63 MB 2025-02-14 21:38:23,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29307.70 MB 2025-02-14 21:38:23,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12712.94 MB 2025-02-14 21:38:23,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25544.16 MB 2025-02-14 21:38:23,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:38:23,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:38:23,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:38:23,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:23,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21565.62 MB 2025-02-14 21:38:23,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23455.15 MB 2025-02-14 21:38:23,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:38:23,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29307.70 MB 2025-02-14 21:38:23,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29307.70 MB 2025-02-14 21:38:23,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:38:23,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24872.58 MB 2025-02-14 21:38:23,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:38:23,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:38:23,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:38:23,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:23,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23455.15 MB 2025-02-14 21:38:23,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25697.01 MB 2025-02-14 21:38:23,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:38:23,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29307.70 MB 2025-02-14 21:38:23,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 21:38:23,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 21:38:23,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31241.29 MB 2025-02-14 21:38:23,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:38:23,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:38:23,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:38:23,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:23,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21565.62 MB 2025-02-14 21:38:23,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25697.01 MB 2025-02-14 21:38:23,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:38:23,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29307.70 MB 2025-02-14 21:38:23,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 21:38:23,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 21:38:23,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31241.29 MB 2025-02-14 21:38:23,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:38:23,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:38:23,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 21:38:23,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:23,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27230.55 MB 2025-02-14 21:38:23,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27997.55 MB 2025-02-14 21:38:23,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:38:23,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 21:38:23,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33499.91 MB 2025-02-14 21:38:23,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:38:23,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28705.34 MB 2025-02-14 21:38:23,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:38:23,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:38:23,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:38:23,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:23,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28410.44 MB 2025-02-14 21:38:23,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28638.88 MB 2025-02-14 21:38:23,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 21:38:23,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33499.91 MB 2025-02-14 21:38:23,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33499.91 MB 2025-02-14 21:38:23,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:38:23,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28861.68 MB 2025-02-14 21:38:23,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:38:23,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:38:23,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.02 seconds 2025-02-14 21:38:23,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:23,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16491.11 MB 2025-02-14 21:38:23,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28839.37 MB 2025-02-14 21:38:23,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12348.25 MB 2025-02-14 21:38:23,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51151.63 MB 2025-02-14 21:38:23,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33499.91 MB 2025-02-14 21:38:23,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17651.73 MB 2025-02-14 21:38:23,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28861.68 MB 2025-02-14 21:38:24,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:38:24,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:38:24,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:38:24,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:24,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28839.37 MB 2025-02-14 21:38:24,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21484.58 MB 2025-02-14 21:38:24,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7354.79 MB 2025-02-14 21:38:24,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33499.91 MB 2025-02-14 21:38:24,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33499.91 MB 2025-02-14 21:38:24,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:38:24,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31342.12 MB 2025-02-14 21:38:24,025 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 21:38:24,025 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:38:24,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:38:24,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:38:24,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:38:24,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:38:24,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21484.58 MB 2025-02-14 21:38:24,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29893.88 MB 2025-02-14 21:38:24,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 21:38:24,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33499.91 MB 2025-02-14 21:38:24,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41859.15 MB 2025-02-14 21:38:24,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 21:38:24,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29893.88 MB 2025-02-14 21:38:24,187 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 21:38:24,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:38:24,188 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:38:24,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:38:24,189 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:38:24,194 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:38:24,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:38:24,195 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:38:24,195 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:39:10,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:39:10,795 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:39:10,800 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:39:10,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:39:10,805 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 207, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:39:10,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:39:10,806 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 207, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:39:14,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:39:14,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:39:14,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.20 seconds 2025-02-14 21:39:14,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:14,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14411.12 MB 2025-02-14 21:39:14,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15143.68 MB 2025-02-14 21:39:14,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 732.56 MB 2025-02-14 21:39:14,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50218.40 MB 2025-02-14 21:39:14,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26524.78 MB 2025-02-14 21:39:14,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23693.62 MB 2025-02-14 21:39:14,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24108.98 MB 2025-02-14 21:39:14,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:39:14,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:39:14,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:39:14,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:14,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15143.68 MB 2025-02-14 21:39:14,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15330.05 MB 2025-02-14 21:39:14,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 186.37 MB 2025-02-14 21:39:14,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26524.78 MB 2025-02-14 21:39:14,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26524.78 MB 2025-02-14 21:39:14,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:39:14,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17767.25 MB 2025-02-14 21:39:14,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:39:14,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:39:14,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.87 seconds 2025-02-14 21:39:14,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:14,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15330.05 MB 2025-02-14 21:39:14,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15572.91 MB 2025-02-14 21:39:14,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 242.86 MB 2025-02-14 21:39:14,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26524.78 MB 2025-02-14 21:39:14,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26052.92 MB 2025-02-14 21:39:14,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 21:39:14,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19499.70 MB 2025-02-14 21:39:14,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:39:14,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:39:14,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:39:14,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:14,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15572.84 MB 2025-02-14 21:39:14,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16437.09 MB 2025-02-14 21:39:14,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 864.25 MB 2025-02-14 21:39:14,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26052.92 MB 2025-02-14 21:39:14,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26052.92 MB 2025-02-14 21:39:14,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:39:14,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17085.57 MB 2025-02-14 21:39:15,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:39:15,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:39:15,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:39:15,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:15,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16437.09 MB 2025-02-14 21:39:15,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17462.78 MB 2025-02-14 21:39:15,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1025.68 MB 2025-02-14 21:39:15,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26052.92 MB 2025-02-14 21:39:15,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26052.92 MB 2025-02-14 21:39:15,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:39:15,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19999.25 MB 2025-02-14 21:39:15,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:39:15,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:39:15,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:39:15,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:15,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15572.84 MB 2025-02-14 21:39:15,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17462.78 MB 2025-02-14 21:39:15,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.94 MB 2025-02-14 21:39:15,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26052.92 MB 2025-02-14 21:39:15,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26052.92 MB 2025-02-14 21:39:15,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:39:15,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19999.25 MB 2025-02-14 21:39:15,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:39:15,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:39:15,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:39:15,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:15,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18164.37 MB 2025-02-14 21:39:15,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18515.28 MB 2025-02-14 21:39:15,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 350.90 MB 2025-02-14 21:39:15,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26052.92 MB 2025-02-14 21:39:15,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26241.66 MB 2025-02-14 21:39:15,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 188.74 MB 2025-02-14 21:39:15,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18845.12 MB 2025-02-14 21:39:15,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:39:15,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:39:15,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:39:15,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:15,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18704.18 MB 2025-02-14 21:39:15,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18929.76 MB 2025-02-14 21:39:15,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.58 MB 2025-02-14 21:39:15,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26241.66 MB 2025-02-14 21:39:15,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26241.66 MB 2025-02-14 21:39:15,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:39:15,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18964.12 MB 2025-02-14 21:39:15,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:39:15,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:39:15,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.29 seconds 2025-02-14 21:39:15,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:15,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13689.91 MB 2025-02-14 21:39:15,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19130.41 MB 2025-02-14 21:39:15,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5440.50 MB 2025-02-14 21:39:15,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50218.40 MB 2025-02-14 21:39:15,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26241.66 MB 2025-02-14 21:39:15,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23976.74 MB 2025-02-14 21:39:15,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19130.41 MB 2025-02-14 21:39:15,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:39:15,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:39:15,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:39:15,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:15,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19130.41 MB 2025-02-14 21:39:15,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17663.74 MB 2025-02-14 21:39:15,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1466.67 MB 2025-02-14 21:39:15,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26241.66 MB 2025-02-14 21:39:15,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26241.66 MB 2025-02-14 21:39:15,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:39:15,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19130.42 MB 2025-02-14 21:39:15,378 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 21:39:15,378 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:39:15,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:39:15,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:39:15,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:39:15,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:39:15,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17663.74 MB 2025-02-14 21:39:15,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26085.70 MB 2025-02-14 21:39:15,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 21:39:15,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26241.66 MB 2025-02-14 21:39:15,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34613.49 MB 2025-02-14 21:39:15,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 21:39:15,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26085.70 MB 2025-02-14 21:39:15,546 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 21:39:15,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:39:15,547 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:39:15,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:39:15,548 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:39:15,553 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:39:15,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:39:15,554 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:39:15,554 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:40:22,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:22,573 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:40:22,578 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:40:22,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:22,583 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1005, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:40:22,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:22,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1005, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:40:37,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:40:37,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:40:37,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.39 seconds 2025-02-14 21:40:37,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:37,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19971.71 MB 2025-02-14 21:40:37,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23528.48 MB 2025-02-14 21:40:37,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3556.77 MB 2025-02-14 21:40:37,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42985.32 MB 2025-02-14 21:40:37,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29137.83 MB 2025-02-14 21:40:37,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13847.49 MB 2025-02-14 21:40:37,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32388.29 MB 2025-02-14 21:40:38,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:40:38,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:40:38,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 21:40:38,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:38,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23528.48 MB 2025-02-14 21:40:38,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21003.58 MB 2025-02-14 21:40:38,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2524.90 MB 2025-02-14 21:40:38,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29137.83 MB 2025-02-14 21:40:38,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40068.19 MB 2025-02-14 21:40:38,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10930.36 MB 2025-02-14 21:40:38,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34859.33 MB 2025-02-14 21:40:39,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:40:39,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:40:39,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 21:40:39,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:39,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21003.58 MB 2025-02-14 21:40:39,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21534.42 MB 2025-02-14 21:40:39,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:40:39,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40068.19 MB 2025-02-14 21:40:39,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27705.48 MB 2025-02-14 21:40:39,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12362.71 MB 2025-02-14 21:40:39,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25512.97 MB 2025-02-14 21:40:39,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:40:39,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:40:39,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:40:39,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:39,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21534.42 MB 2025-02-14 21:40:39,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23423.96 MB 2025-02-14 21:40:39,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:40:39,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27705.48 MB 2025-02-14 21:40:39,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27705.48 MB 2025-02-14 21:40:39,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:40:39,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24841.38 MB 2025-02-14 21:40:40,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:40:40,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:40:40,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:40:40,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:40,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23423.96 MB 2025-02-14 21:40:40,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25665.81 MB 2025-02-14 21:40:40,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:40:40,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27705.48 MB 2025-02-14 21:40:40,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33367.79 MB 2025-02-14 21:40:40,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:40:40,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31210.09 MB 2025-02-14 21:40:40,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:40:40,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:40:40,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:40:40,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:40,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21534.42 MB 2025-02-14 21:40:40,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25665.81 MB 2025-02-14 21:40:40,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:40:40,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27705.48 MB 2025-02-14 21:40:40,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33367.79 MB 2025-02-14 21:40:40,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:40:40,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31210.09 MB 2025-02-14 21:40:40,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:40:40,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:40:40,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:40:40,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:40,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27199.35 MB 2025-02-14 21:40:40,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27966.36 MB 2025-02-14 21:40:40,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:40:40,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33367.79 MB 2025-02-14 21:40:40,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33785.12 MB 2025-02-14 21:40:40,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:40:40,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28674.14 MB 2025-02-14 21:40:40,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:40:40,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:40:40,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:40:40,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:40,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28379.25 MB 2025-02-14 21:40:40,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28606.18 MB 2025-02-14 21:40:40,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.93 MB 2025-02-14 21:40:40,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33785.12 MB 2025-02-14 21:40:40,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33785.12 MB 2025-02-14 21:40:40,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:40:40,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28844.08 MB 2025-02-14 21:40:40,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:40:40,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:40:40,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.77 seconds 2025-02-14 21:40:40,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:40,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16470.21 MB 2025-02-14 21:40:40,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28806.12 MB 2025-02-14 21:40:40,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12335.91 MB 2025-02-14 21:40:40,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42985.32 MB 2025-02-14 21:40:40,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33785.12 MB 2025-02-14 21:40:40,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9200.21 MB 2025-02-14 21:40:40,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28844.08 MB 2025-02-14 21:40:40,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:40:40,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:40:40,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:40:40,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:40,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28806.12 MB 2025-02-14 21:40:40,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21457.61 MB 2025-02-14 21:40:40,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7348.51 MB 2025-02-14 21:40:40,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33785.12 MB 2025-02-14 21:40:40,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33785.12 MB 2025-02-14 21:40:40,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:40:40,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31304.20 MB 2025-02-14 21:40:40,645 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 21:40:40,645 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:40:40,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:40:40,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:40:40,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:40:40,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:40:40,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21457.61 MB 2025-02-14 21:40:40,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29850.04 MB 2025-02-14 21:40:40,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 21:40:40,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33785.12 MB 2025-02-14 21:40:40,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42127.59 MB 2025-02-14 21:40:40,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 21:40:40,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29850.04 MB 2025-02-14 21:40:40,808 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 21:40:40,809 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:40,809 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:40:40,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:40,810 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:40:40,815 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:40:40,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:40,816 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:40:40,816 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:40:58,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:58,261 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:40:58,266 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:40:58,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:58,269 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1655, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:40:58,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:40:58,270 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1655, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:41:23,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:41:23,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:41:23,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.52 seconds 2025-02-14 21:41:23,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:23,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24501.01 MB 2025-02-14 21:41:23,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30358.36 MB 2025-02-14 21:41:23,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5857.35 MB 2025-02-14 21:41:23,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50470.06 MB 2025-02-14 21:41:23,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35590.77 MB 2025-02-14 21:41:23,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14879.29 MB 2025-02-14 21:41:23,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39182.51 MB 2025-02-14 21:41:23,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:41:23,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:41:23,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:41:23,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:23,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30358.36 MB 2025-02-14 21:41:23,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24381.68 MB 2025-02-14 21:41:23,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5976.68 MB 2025-02-14 21:41:23,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35590.77 MB 2025-02-14 21:41:23,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56669.24 MB 2025-02-14 21:41:23,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21078.47 MB 2025-02-14 21:41:23,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47718.36 MB 2025-02-14 21:41:25,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:41:25,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:41:25,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 21:41:25,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:25,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24381.68 MB 2025-02-14 21:41:25,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24912.52 MB 2025-02-14 21:41:25,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:41:25,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56669.24 MB 2025-02-14 21:41:25,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27873.25 MB 2025-02-14 21:41:25,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28795.99 MB 2025-02-14 21:41:25,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.11 MB 2025-02-14 21:41:25,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:41:25,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:41:25,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:41:25,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:25,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24912.52 MB 2025-02-14 21:41:25,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26802.06 MB 2025-02-14 21:41:25,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:41:25,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 21:41:25,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29760.68 MB 2025-02-14 21:41:25,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:41:25,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28219.48 MB 2025-02-14 21:41:26,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:41:26,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:41:26,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:41:26,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:26,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26802.06 MB 2025-02-14 21:41:26,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29043.91 MB 2025-02-14 21:41:26,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:41:26,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29760.68 MB 2025-02-14 21:41:26,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36366.71 MB 2025-02-14 21:41:26,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 21:41:26,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.19 MB 2025-02-14 21:41:26,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:41:26,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:41:26,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 21:41:26,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:26,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24912.52 MB 2025-02-14 21:41:26,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29043.91 MB 2025-02-14 21:41:26,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:41:26,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 21:41:26,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36366.71 MB 2025-02-14 21:41:26,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 21:41:26,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.19 MB 2025-02-14 21:41:26,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:41:26,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:41:26,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:41:26,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:26,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30577.45 MB 2025-02-14 21:41:26,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31344.46 MB 2025-02-14 21:41:26,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:41:26,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36366.71 MB 2025-02-14 21:41:26,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36781.95 MB 2025-02-14 21:41:26,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:41:26,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32052.24 MB 2025-02-14 21:41:26,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:41:26,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:41:26,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:41:26,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:26,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31757.34 MB 2025-02-14 21:41:26,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31986.21 MB 2025-02-14 21:41:26,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-14 21:41:26,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36781.95 MB 2025-02-14 21:41:26,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36781.95 MB 2025-02-14 21:41:26,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:41:26,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32234.70 MB 2025-02-14 21:41:26,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:41:26,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:41:26,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.02 seconds 2025-02-14 21:41:26,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:26,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18734.86 MB 2025-02-14 21:41:26,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32186.98 MB 2025-02-14 21:41:26,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13452.13 MB 2025-02-14 21:41:26,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50470.06 MB 2025-02-14 21:41:26,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36781.95 MB 2025-02-14 21:41:26,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13688.11 MB 2025-02-14 21:41:26,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32234.70 MB 2025-02-14 21:41:26,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:41:26,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:41:26,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:41:26,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:26,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32186.98 MB 2025-02-14 21:41:26,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23734.68 MB 2025-02-14 21:41:26,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8452.31 MB 2025-02-14 21:41:26,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36781.95 MB 2025-02-14 21:41:26,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36781.95 MB 2025-02-14 21:41:26,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:41:26,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34694.97 MB 2025-02-14 21:41:26,574 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 21:41:26,574 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:41:26,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:41:26,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:41:26,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:41:26,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:26,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23734.68 MB 2025-02-14 21:41:26,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32161.18 MB 2025-02-14 21:41:26,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 21:41:26,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36781.95 MB 2025-02-14 21:41:26,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45160.07 MB 2025-02-14 21:41:26,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 21:41:26,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32161.18 MB 2025-02-14 21:41:26,741 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 21:41:26,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:26,742 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:41:26,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:26,743 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:41:26,748 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:41:26,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:26,749 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:41:26,749 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:41:38,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:38,107 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:41:38,112 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:41:38,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:38,116 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:41:38,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:38,117 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:41:44,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:41:44,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:41:44,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.93 seconds 2025-02-14 21:41:44,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:44,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-14 21:41:44,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16950.90 MB 2025-02-14 21:41:44,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1341.26 MB 2025-02-14 21:41:44,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57726.21 MB 2025-02-14 21:41:44,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25165.82 MB 2025-02-14 21:41:44,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32560.38 MB 2025-02-14 21:41:44,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25760.49 MB 2025-02-14 21:41:44,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:41:44,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:41:44,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:41:44,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:44,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16950.90 MB 2025-02-14 21:41:44,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17530.44 MB 2025-02-14 21:41:44,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 579.54 MB 2025-02-14 21:41:44,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25165.82 MB 2025-02-14 21:41:44,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25165.82 MB 2025-02-14 21:41:44,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:41:44,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22172.82 MB 2025-02-14 21:41:45,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:41:45,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:41:45,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.78 seconds 2025-02-14 21:41:45,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:45,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17530.44 MB 2025-02-14 21:41:45,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18020.14 MB 2025-02-14 21:41:45,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 489.70 MB 2025-02-14 21:41:45,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25165.82 MB 2025-02-14 21:41:45,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20946.35 MB 2025-02-14 21:41:45,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4219.47 MB 2025-02-14 21:41:45,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21955.93 MB 2025-02-14 21:41:45,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:41:45,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:41:45,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:41:45,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:45,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18020.14 MB 2025-02-14 21:41:45,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19763.14 MB 2025-02-14 21:41:45,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1743.00 MB 2025-02-14 21:41:45,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20946.35 MB 2025-02-14 21:41:45,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23563.60 MB 2025-02-14 21:41:45,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2617.25 MB 2025-02-14 21:41:45,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21070.72 MB 2025-02-14 21:41:46,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:41:46,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:41:46,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 21:41:46,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:46,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19763.14 MB 2025-02-14 21:41:46,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21831.25 MB 2025-02-14 21:41:46,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2068.12 MB 2025-02-14 21:41:46,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23563.60 MB 2025-02-14 21:41:46,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29234.30 MB 2025-02-14 21:41:46,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5670.70 MB 2025-02-14 21:41:46,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26946.77 MB 2025-02-14 21:41:46,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:41:46,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:41:46,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:41:46,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:46,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18020.14 MB 2025-02-14 21:41:46,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21831.25 MB 2025-02-14 21:41:46,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3811.11 MB 2025-02-14 21:41:46,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20946.35 MB 2025-02-14 21:41:46,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29234.30 MB 2025-02-14 21:41:46,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8287.94 MB 2025-02-14 21:41:46,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26946.77 MB 2025-02-14 21:41:46,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:41:46,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:41:46,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 21:41:46,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:46,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23245.95 MB 2025-02-14 21:41:46,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23954.42 MB 2025-02-14 21:41:46,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 708.48 MB 2025-02-14 21:41:46,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29234.30 MB 2025-02-14 21:41:46,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29615.98 MB 2025-02-14 21:41:46,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 381.68 MB 2025-02-14 21:41:46,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24607.36 MB 2025-02-14 21:41:46,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:41:46,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:41:46,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:41:46,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:46,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24335.32 MB 2025-02-14 21:41:46,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24563.22 MB 2025-02-14 21:41:46,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.91 MB 2025-02-14 21:41:46,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29615.98 MB 2025-02-14 21:41:46,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29615.98 MB 2025-02-14 21:41:46,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:41:46,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24717.03 MB 2025-02-14 21:41:46,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:41:46,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:41:46,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.11 seconds 2025-02-14 21:41:46,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:46,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-14 21:41:46,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24764.30 MB 2025-02-14 21:41:46,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10475.12 MB 2025-02-14 21:41:46,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57726.21 MB 2025-02-14 21:41:46,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29615.98 MB 2025-02-14 21:41:46,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28110.23 MB 2025-02-14 21:41:46,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24764.30 MB 2025-02-14 21:41:46,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:41:46,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:41:46,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:41:46,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:46,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24764.30 MB 2025-02-14 21:41:46,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19147.92 MB 2025-02-14 21:41:46,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5616.38 MB 2025-02-14 21:41:46,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29615.98 MB 2025-02-14 21:41:46,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29615.98 MB 2025-02-14 21:41:46,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:41:46,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27778.30 MB 2025-02-14 21:41:46,513 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:41:46,514 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:41:46,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:41:46,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:41:46,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:41:46,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:41:46,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19147.92 MB 2025-02-14 21:41:46,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27586.94 MB 2025-02-14 21:41:46,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:41:46,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29615.98 MB 2025-02-14 21:41:46,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38006.69 MB 2025-02-14 21:41:46,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:41:46,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27586.94 MB 2025-02-14 21:41:46,676 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:41:46,677 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:46,677 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:41:46,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:46,678 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:41:46,683 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:41:46,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:41:46,684 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:41:46,684 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:42:42,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:42:42,938 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:42:42,943 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:42:42,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:42:42,947 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 223, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:42:42,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:42:42,948 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 223, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:42:46,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:42:46,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:42:46,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.44 seconds 2025-02-14 21:42:46,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:46,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14522.61 MB 2025-02-14 21:42:46,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.79 MB 2025-02-14 21:42:46,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 789.18 MB 2025-02-14 21:42:46,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50591.69 MB 2025-02-14 21:42:46,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20296.24 MB 2025-02-14 21:42:46,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30295.46 MB 2025-02-14 21:42:46,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24220.47 MB 2025-02-14 21:42:46,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:42:46,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:42:46,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:42:46,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:46,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.79 MB 2025-02-14 21:42:46,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15595.76 MB 2025-02-14 21:42:46,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.97 MB 2025-02-14 21:42:46,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20296.24 MB 2025-02-14 21:42:46,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20296.24 MB 2025-02-14 21:42:46,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:42:46,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18300.50 MB 2025-02-14 21:42:47,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:42:47,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:42:47,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-14 21:42:47,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15595.76 MB 2025-02-14 21:42:47,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15873.13 MB 2025-02-14 21:42:47,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-14 21:42:47,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20296.24 MB 2025-02-14 21:42:47,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20698.89 MB 2025-02-14 21:42:47,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 21:42:47,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19851.89 MB 2025-02-14 21:42:47,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:42:47,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:42:47,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:42:47,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.13 MB 2025-02-14 21:42:47,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16860.17 MB 2025-02-14 21:42:47,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-14 21:42:47,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20698.89 MB 2025-02-14 21:42:47,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20698.89 MB 2025-02-14 21:42:47,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:42:47,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17600.78 MB 2025-02-14 21:42:47,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:42:47,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:42:47,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 21:42:47,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16860.17 MB 2025-02-14 21:42:47,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18031.57 MB 2025-02-14 21:42:47,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.40 MB 2025-02-14 21:42:47,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20698.89 MB 2025-02-14 21:42:47,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22431.14 MB 2025-02-14 21:42:47,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1732.25 MB 2025-02-14 21:42:47,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20929.34 MB 2025-02-14 21:42:47,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:42:47,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:42:47,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 21:42:47,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.13 MB 2025-02-14 21:42:47,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18031.57 MB 2025-02-14 21:42:47,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.44 MB 2025-02-14 21:42:47,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20698.89 MB 2025-02-14 21:42:47,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22431.14 MB 2025-02-14 21:42:47,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1732.25 MB 2025-02-14 21:42:47,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20929.34 MB 2025-02-14 21:42:47,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:42:47,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:42:47,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:42:47,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18832.84 MB 2025-02-14 21:42:47,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19234.52 MB 2025-02-14 21:42:47,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 401.68 MB 2025-02-14 21:42:47,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22431.14 MB 2025-02-14 21:42:47,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22647.14 MB 2025-02-14 21:42:47,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-14 21:42:47,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19606.76 MB 2025-02-14 21:42:47,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:42:47,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:42:47,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:42:47,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19450.26 MB 2025-02-14 21:42:47,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19679.79 MB 2025-02-14 21:42:47,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.53 MB 2025-02-14 21:42:47,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22647.14 MB 2025-02-14 21:42:47,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22647.14 MB 2025-02-14 21:42:47,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:42:47,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19733.54 MB 2025-02-14 21:42:47,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:42:47,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:42:47,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.69 seconds 2025-02-14 21:42:47,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13745.66 MB 2025-02-14 21:42:47,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19880.86 MB 2025-02-14 21:42:47,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6135.21 MB 2025-02-14 21:42:47,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50591.69 MB 2025-02-14 21:42:47,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22647.14 MB 2025-02-14 21:42:47,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27944.55 MB 2025-02-14 21:42:47,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.86 MB 2025-02-14 21:42:47,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:42:47,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:42:47,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:42:47,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14835.46 MB 2025-02-14 21:42:47,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17849.50 MB 2025-02-14 21:42:47,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 21:42:47,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22647.14 MB 2025-02-14 21:42:47,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22647.14 MB 2025-02-14 21:42:47,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:42:47,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18150.86 MB 2025-02-14 21:42:47,921 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:42:47,922 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 21:42:47,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:42:47,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:42:47,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:42:47,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:42:47,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17849.50 MB 2025-02-14 21:42:47,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26288.52 MB 2025-02-14 21:42:47,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:42:47,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22647.14 MB 2025-02-14 21:42:47,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31037.85 MB 2025-02-14 21:42:47,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:42:47,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26288.52 MB 2025-02-14 21:42:48,087 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:42:48,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:42:48,089 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:42:48,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:42:48,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:42:48,094 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:42:48,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:42:48,095 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:42:48,096 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 21:43:06,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:06,937 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:43:06,942 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:43:06,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:06,945 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:43:06,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:06,946 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:43:25,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:43:25,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:43:25,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.55 seconds 2025-02-14 21:43:25,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:25,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-14 21:43:25,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.80 MB 2025-02-14 21:43:25,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4278.58 MB 2025-02-14 21:43:25,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43622.86 MB 2025-02-14 21:43:25,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30784.09 MB 2025-02-14 21:43:25,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12838.76 MB 2025-02-14 21:43:25,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34488.46 MB 2025-02-14 21:43:25,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:43:25,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:43:25,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:43:25,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:25,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.80 MB 2025-02-14 21:43:25,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22064.11 MB 2025-02-14 21:43:25,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3607.68 MB 2025-02-14 21:43:25,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30784.09 MB 2025-02-14 21:43:25,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44568.67 MB 2025-02-14 21:43:25,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13784.58 MB 2025-02-14 21:43:25,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38168.08 MB 2025-02-14 21:43:27,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:43:27,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:43:27,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:43:27,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:27,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22064.11 MB 2025-02-14 21:43:27,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22594.96 MB 2025-02-14 21:43:27,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:43:27,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44568.67 MB 2025-02-14 21:43:27,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24433.92 MB 2025-02-14 21:43:27,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20134.76 MB 2025-02-14 21:43:27,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26574.54 MB 2025-02-14 21:43:27,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:43:27,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:43:27,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:43:27,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:27,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22594.96 MB 2025-02-14 21:43:27,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24484.49 MB 2025-02-14 21:43:27,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:43:27,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24433.92 MB 2025-02-14 21:43:27,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27265.07 MB 2025-02-14 21:43:27,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 21:43:27,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25901.92 MB 2025-02-14 21:43:27,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:43:27,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:43:27,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:43:27,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:27,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24484.49 MB 2025-02-14 21:43:27,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26726.35 MB 2025-02-14 21:43:27,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:43:27,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27265.07 MB 2025-02-14 21:43:27,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33871.10 MB 2025-02-14 21:43:27,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 21:43:27,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32270.63 MB 2025-02-14 21:43:27,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:43:27,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:43:27,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:43:27,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:27,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22594.96 MB 2025-02-14 21:43:27,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26726.35 MB 2025-02-14 21:43:27,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:43:27,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24433.92 MB 2025-02-14 21:43:27,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33871.10 MB 2025-02-14 21:43:27,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 21:43:27,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32270.63 MB 2025-02-14 21:43:27,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:43:27,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:43:27,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 21:43:27,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:27,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28259.89 MB 2025-02-14 21:43:27,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29026.89 MB 2025-02-14 21:43:27,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:43:27,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33871.10 MB 2025-02-14 21:43:27,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-14 21:43:27,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:43:27,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29734.68 MB 2025-02-14 21:43:27,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:43:27,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:43:27,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:43:27,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:27,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29439.78 MB 2025-02-14 21:43:27,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29668.64 MB 2025-02-14 21:43:27,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-14 21:43:27,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34286.34 MB 2025-02-14 21:43:27,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-14 21:43:27,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:43:27,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29899.86 MB 2025-02-14 21:43:27,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:43:27,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:43:27,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.99 seconds 2025-02-14 21:43:27,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:27,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17180.96 MB 2025-02-14 21:43:27,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29869.42 MB 2025-02-14 21:43:27,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12688.46 MB 2025-02-14 21:43:27,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43622.86 MB 2025-02-14 21:43:27,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-14 21:43:27,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9336.52 MB 2025-02-14 21:43:27,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29899.86 MB 2025-02-14 21:43:28,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:43:28,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:43:28,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:43:28,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:28,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29869.42 MB 2025-02-14 21:43:28,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22180.78 MB 2025-02-14 21:43:28,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7688.64 MB 2025-02-14 21:43:28,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34286.34 MB 2025-02-14 21:43:28,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-14 21:43:28,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:43:28,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32377.40 MB 2025-02-14 21:43:28,223 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 21:43:28,223 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 21:43:28,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:43:28,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:43:28,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:43:28,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:28,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22180.78 MB 2025-02-14 21:43:28,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30607.28 MB 2025-02-14 21:43:28,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 21:43:28,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34286.34 MB 2025-02-14 21:43:28,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42664.46 MB 2025-02-14 21:43:28,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 21:43:28,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30607.28 MB 2025-02-14 21:43:28,386 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 21:43:28,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:28,387 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:43:28,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:28,388 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:43:28,393 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:43:28,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:28,394 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:43:28,394 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 21:43:40,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:40,021 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:43:40,028 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:43:40,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:40,035 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 414, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:43:40,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:40,037 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 414, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:43:46,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:43:46,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:43:46,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.50 seconds 2025-02-14 21:43:46,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:46,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15853.52 MB 2025-02-14 21:43:46,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17318.65 MB 2025-02-14 21:43:46,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1465.12 MB 2025-02-14 21:43:46,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55230.60 MB 2025-02-14 21:43:46,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25163.73 MB 2025-02-14 21:43:46,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30066.87 MB 2025-02-14 21:43:46,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26230.87 MB 2025-02-14 21:43:46,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:43:46,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:43:46,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:43:46,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:46,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17318.65 MB 2025-02-14 21:43:46,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17621.09 MB 2025-02-14 21:43:46,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 302.45 MB 2025-02-14 21:43:46,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25163.73 MB 2025-02-14 21:43:46,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25163.73 MB 2025-02-14 21:43:46,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:43:46,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22368.59 MB 2025-02-14 21:43:48,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:43:48,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:43:48,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.70 seconds 2025-02-14 21:43:48,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17621.09 MB 2025-02-14 21:43:48,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18093.54 MB 2025-02-14 21:43:48,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 472.45 MB 2025-02-14 21:43:48,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25163.73 MB 2025-02-14 21:43:48,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24220.01 MB 2025-02-14 21:43:48,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 21:43:48,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22045.55 MB 2025-02-14 21:43:48,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:43:48,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:43:48,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:43:48,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18093.54 MB 2025-02-14 21:43:48,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19775.46 MB 2025-02-14 21:43:48,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1681.92 MB 2025-02-14 21:43:48,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24220.01 MB 2025-02-14 21:43:48,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24220.01 MB 2025-02-14 21:43:48,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:43:48,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21036.97 MB 2025-02-14 21:43:48,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:43:48,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:43:48,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 21:43:48,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19775.46 MB 2025-02-14 21:43:48,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21770.72 MB 2025-02-14 21:43:48,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1995.26 MB 2025-02-14 21:43:48,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24220.01 MB 2025-02-14 21:43:48,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28848.42 MB 2025-02-14 21:43:48,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4628.41 MB 2025-02-14 21:43:48,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26709.32 MB 2025-02-14 21:43:48,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:43:48,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:43:48,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 21:43:48,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18093.54 MB 2025-02-14 21:43:48,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21770.72 MB 2025-02-14 21:43:48,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3677.18 MB 2025-02-14 21:43:48,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24220.01 MB 2025-02-14 21:43:48,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28848.42 MB 2025-02-14 21:43:48,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4628.41 MB 2025-02-14 21:43:48,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26709.32 MB 2025-02-14 21:43:48,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:43:48,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:43:48,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 21:43:48,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23135.57 MB 2025-02-14 21:43:48,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23818.20 MB 2025-02-14 21:43:48,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 682.63 MB 2025-02-14 21:43:48,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28848.42 MB 2025-02-14 21:43:48,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29219.62 MB 2025-02-14 21:43:48,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 371.20 MB 2025-02-14 21:43:48,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24448.14 MB 2025-02-14 21:43:48,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:43:48,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:43:48,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:43:48,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24185.68 MB 2025-02-14 21:43:48,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24398.97 MB 2025-02-14 21:43:48,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.30 MB 2025-02-14 21:43:48,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29219.62 MB 2025-02-14 21:43:48,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29219.62 MB 2025-02-14 21:43:48,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:43:48,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24549.25 MB 2025-02-14 21:43:48,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:43:48,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:43:48,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.59 seconds 2025-02-14 21:43:48,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14411.12 MB 2025-02-14 21:43:48,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24600.05 MB 2025-02-14 21:43:48,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10188.93 MB 2025-02-14 21:43:48,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55230.60 MB 2025-02-14 21:43:48,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29219.62 MB 2025-02-14 21:43:48,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26010.98 MB 2025-02-14 21:43:48,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24600.05 MB 2025-02-14 21:43:48,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:43:48,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:43:48,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:43:48,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24600.05 MB 2025-02-14 21:43:48,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19207.33 MB 2025-02-14 21:43:48,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5392.71 MB 2025-02-14 21:43:48,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29219.62 MB 2025-02-14 21:43:48,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29219.62 MB 2025-02-14 21:43:48,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:43:48,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27814.98 MB 2025-02-14 21:43:48,914 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:43:48,914 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 21:43:48,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:43:48,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:43:48,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:43:48,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:43:48,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19207.33 MB 2025-02-14 21:43:48,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27646.36 MB 2025-02-14 21:43:48,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:43:48,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29219.62 MB 2025-02-14 21:43:48,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37610.32 MB 2025-02-14 21:43:48,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:43:48,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27646.36 MB 2025-02-14 21:43:49,076 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:43:49,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:49,078 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:43:49,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:49,079 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:43:49,083 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:43:49,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:43:49,084 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:43:49,084 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 21:44:06,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:44:06,743 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:44:06,751 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:44:06,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:44:06,758 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 191, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:44:06,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:44:06,760 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 191, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:44:09,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:44:09,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:44:09,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.00 seconds 2025-02-14 21:44:09,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:09,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14299.63 MB 2025-02-14 21:44:09,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14975.56 MB 2025-02-14 21:44:09,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.94 MB 2025-02-14 21:44:09,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50195.33 MB 2025-02-14 21:44:09,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19801.31 MB 2025-02-14 21:44:09,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30394.02 MB 2025-02-14 21:44:09,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23853.38 MB 2025-02-14 21:44:09,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:44:09,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:44:09,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:44:09,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:09,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14975.56 MB 2025-02-14 21:44:09,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15204.73 MB 2025-02-14 21:44:09,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-14 21:44:09,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19801.31 MB 2025-02-14 21:44:09,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19801.31 MB 2025-02-14 21:44:09,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:44:09,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17514.86 MB 2025-02-14 21:44:10,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:44:10,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:44:10,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 21:44:10,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:10,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15204.73 MB 2025-02-14 21:44:10,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15439.63 MB 2025-02-14 21:44:10,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 21:44:10,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19801.31 MB 2025-02-14 21:44:10,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19801.31 MB 2025-02-14 21:44:10,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:44:10,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19375.42 MB 2025-02-14 21:44:10,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:44:10,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:44:10,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:44:10,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:10,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15439.56 MB 2025-02-14 21:44:10,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16275.48 MB 2025-02-14 21:44:10,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 21:44:10,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19801.31 MB 2025-02-14 21:44:10,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19801.31 MB 2025-02-14 21:44:10,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:44:10,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16902.70 MB 2025-02-14 21:44:10,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:44:10,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:44:10,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:44:10,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:10,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16275.48 MB 2025-02-14 21:44:10,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17267.54 MB 2025-02-14 21:44:10,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 21:44:10,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19801.31 MB 2025-02-14 21:44:10,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21269.32 MB 2025-02-14 21:44:10,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1468.01 MB 2025-02-14 21:44:10,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19721.76 MB 2025-02-14 21:44:10,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:44:10,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:44:10,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:44:10,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:10,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15439.56 MB 2025-02-14 21:44:10,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17267.54 MB 2025-02-14 21:44:10,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 21:44:10,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19801.31 MB 2025-02-14 21:44:10,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21269.32 MB 2025-02-14 21:44:10,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1468.01 MB 2025-02-14 21:44:10,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19721.76 MB 2025-02-14 21:44:10,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:44:10,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:44:10,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:44:10,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:10,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17946.13 MB 2025-02-14 21:44:10,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18286.45 MB 2025-02-14 21:44:10,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-14 21:44:10,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21269.32 MB 2025-02-14 21:44:10,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21449.67 MB 2025-02-14 21:44:10,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 21:44:10,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18605.78 MB 2025-02-14 21:44:10,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:44:10,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:44:10,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:44:10,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:10,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18469.16 MB 2025-02-14 21:44:10,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18696.71 MB 2025-02-14 21:44:10,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.55 MB 2025-02-14 21:44:10,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21449.67 MB 2025-02-14 21:44:10,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21449.67 MB 2025-02-14 21:44:10,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:44:10,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18721.14 MB 2025-02-14 21:44:10,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:44:10,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:44:10,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.05 seconds 2025-02-14 21:44:10,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:10,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13634.17 MB 2025-02-14 21:44:10,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18897.78 MB 2025-02-14 21:44:10,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5263.62 MB 2025-02-14 21:44:10,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50195.33 MB 2025-02-14 21:44:10,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21449.67 MB 2025-02-14 21:44:10,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28745.66 MB 2025-02-14 21:44:10,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18897.78 MB 2025-02-14 21:44:11,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:44:11,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:44:11,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:44:11,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:11,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18897.78 MB 2025-02-14 21:44:11,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17586.99 MB 2025-02-14 21:44:11,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1310.79 MB 2025-02-14 21:44:11,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21449.67 MB 2025-02-14 21:44:11,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21449.67 MB 2025-02-14 21:44:11,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:44:11,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19132.21 MB 2025-02-14 21:44:11,101 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:44:11,101 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:44:11,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:44:11,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:44:11,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:44:11,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:44:11,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17586.99 MB 2025-02-14 21:44:11,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26026.02 MB 2025-02-14 21:44:11,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:44:11,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21449.67 MB 2025-02-14 21:44:11,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29840.38 MB 2025-02-14 21:44:11,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:44:11,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26026.02 MB 2025-02-14 21:44:11,262 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:44:11,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:44:11,263 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:44:11,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:44:11,264 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:44:11,269 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:44:11,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:44:11,270 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:44:11,270 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:45:11,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:45:11,661 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:45:11,666 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:45:11,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:45:11,670 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 366, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:45:11,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:45:11,671 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 366, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:45:17,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:45:17,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:45:17,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.59 seconds 2025-02-14 21:45:17,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:17,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15519.05 MB 2025-02-14 21:45:17,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16814.31 MB 2025-02-14 21:45:17,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1295.25 MB 2025-02-14 21:45:17,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42425.38 MB 2025-02-14 21:45:17,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18956.16 MB 2025-02-14 21:45:17,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23469.23 MB 2025-02-14 21:45:17,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25669.90 MB 2025-02-14 21:45:17,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:45:17,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:45:17,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:45:17,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:17,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16814.31 MB 2025-02-14 21:45:17,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17161.84 MB 2025-02-14 21:45:17,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.54 MB 2025-02-14 21:45:17,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18956.16 MB 2025-02-14 21:45:17,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23389.54 MB 2025-02-14 21:45:17,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4433.38 MB 2025-02-14 21:45:17,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21444.82 MB 2025-02-14 21:45:18,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:45:18,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:45:18,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.57 seconds 2025-02-14 21:45:18,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:18,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17161.84 MB 2025-02-14 21:45:18,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17594.48 MB 2025-02-14 21:45:18,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.64 MB 2025-02-14 21:45:18,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23389.54 MB 2025-02-14 21:45:18,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20671.63 MB 2025-02-14 21:45:18,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2717.91 MB 2025-02-14 21:45:18,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21587.34 MB 2025-02-14 21:45:18,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:45:18,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:45:18,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:45:18,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:18,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17594.48 MB 2025-02-14 21:45:18,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19135.62 MB 2025-02-14 21:45:18,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1541.14 MB 2025-02-14 21:45:18,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20671.63 MB 2025-02-14 21:45:18,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22210.94 MB 2025-02-14 21:45:18,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1539.31 MB 2025-02-14 21:45:18,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20290.83 MB 2025-02-14 21:45:19,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:45:19,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:45:19,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 21:45:19,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:19,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19135.62 MB 2025-02-14 21:45:19,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20963.27 MB 2025-02-14 21:45:19,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.65 MB 2025-02-14 21:45:19,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22210.94 MB 2025-02-14 21:45:19,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27214.74 MB 2025-02-14 21:45:19,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5003.80 MB 2025-02-14 21:45:19,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25483.95 MB 2025-02-14 21:45:19,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:45:19,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:45:19,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 21:45:19,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:19,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17594.48 MB 2025-02-14 21:45:19,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20963.27 MB 2025-02-14 21:45:19,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3368.79 MB 2025-02-14 21:45:19,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20671.63 MB 2025-02-14 21:45:19,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27214.74 MB 2025-02-14 21:45:19,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6543.11 MB 2025-02-14 21:45:19,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25483.95 MB 2025-02-14 21:45:19,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:45:19,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:45:19,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 21:45:19,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:19,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22213.11 MB 2025-02-14 21:45:19,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22838.22 MB 2025-02-14 21:45:19,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 625.11 MB 2025-02-14 21:45:19,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27214.74 MB 2025-02-14 21:45:19,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27552.38 MB 2025-02-14 21:45:19,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 337.64 MB 2025-02-14 21:45:19,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23415.07 MB 2025-02-14 21:45:19,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:45:19,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:45:19,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:45:19,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:19,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23174.72 MB 2025-02-14 21:45:19,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23393.15 MB 2025-02-14 21:45:19,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.43 MB 2025-02-14 21:45:19,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27552.38 MB 2025-02-14 21:45:19,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27552.38 MB 2025-02-14 21:45:19,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:45:19,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23510.60 MB 2025-02-14 21:45:19,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:45:19,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:45:19,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.52 seconds 2025-02-14 21:45:19,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:19,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14243.88 MB 2025-02-14 21:45:19,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23594.23 MB 2025-02-14 21:45:19,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9350.35 MB 2025-02-14 21:45:19,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42425.38 MB 2025-02-14 21:45:19,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27552.38 MB 2025-02-14 21:45:19,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14873.00 MB 2025-02-14 21:45:19,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23594.23 MB 2025-02-14 21:45:19,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:45:19,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:45:19,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:45:19,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:19,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23594.23 MB 2025-02-14 21:45:19,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26608.26 MB 2025-02-14 21:45:19,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 21:45:19,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27552.38 MB 2025-02-14 21:45:19,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27955.04 MB 2025-02-14 21:45:19,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 21:45:19,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26909.89 MB 2025-02-14 21:45:19,481 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:45:19,481 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 21:45:19,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:45:19,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:45:19,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:45:19,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:45:19,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18899.04 MB 2025-02-14 21:45:19,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.06 MB 2025-02-14 21:45:19,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:45:19,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27955.04 MB 2025-02-14 21:45:19,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38444.99 MB 2025-02-14 21:45:19,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 21:45:19,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27338.06 MB 2025-02-14 21:45:19,644 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:45:19,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:45:19,646 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:45:19,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:45:19,647 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:45:19,651 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:45:19,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:45:19,652 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:45:19,653 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 21:47:12,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:47:12,832 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:47:12,837 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:47:12,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:47:12,840 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1376, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:47:12,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:47:12,841 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1376, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:47:33,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:47:33,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:47:33,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.02 seconds 2025-02-14 21:47:33,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:33,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22556.89 MB 2025-02-14 21:47:33,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27426.48 MB 2025-02-14 21:47:33,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4869.59 MB 2025-02-14 21:47:33,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51030.00 MB 2025-02-14 21:47:33,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35565.60 MB 2025-02-14 21:47:33,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15464.40 MB 2025-02-14 21:47:33,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36331.62 MB 2025-02-14 21:47:33,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:47:33,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:47:33,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:47:33,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:33,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27426.48 MB 2025-02-14 21:47:33,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22931.25 MB 2025-02-14 21:47:33,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4495.24 MB 2025-02-14 21:47:33,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35565.60 MB 2025-02-14 21:47:33,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46848.28 MB 2025-02-14 21:47:33,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11282.68 MB 2025-02-14 21:47:33,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41849.69 MB 2025-02-14 21:47:35,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:47:35,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:47:35,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 21:47:35,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:35,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22931.25 MB 2025-02-14 21:47:35,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23462.09 MB 2025-02-14 21:47:35,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:47:35,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46848.28 MB 2025-02-14 21:47:35,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30696.01 MB 2025-02-14 21:47:35,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16152.26 MB 2025-02-14 21:47:35,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27440.63 MB 2025-02-14 21:47:35,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:47:35,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:47:35,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:47:35,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:35,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23462.09 MB 2025-02-14 21:47:35,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25351.62 MB 2025-02-14 21:47:35,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:47:35,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 21:47:35,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30696.01 MB 2025-02-14 21:47:35,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:47:35,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26769.05 MB 2025-02-14 21:47:36,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:47:36,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:47:36,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 21:47:36,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:36,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25351.62 MB 2025-02-14 21:47:36,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27593.48 MB 2025-02-14 21:47:36,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:47:36,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 21:47:36,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35414.61 MB 2025-02-14 21:47:36,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:47:36,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33137.76 MB 2025-02-14 21:47:36,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:47:36,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:47:36,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 21:47:36,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:36,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23462.09 MB 2025-02-14 21:47:36,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27593.48 MB 2025-02-14 21:47:36,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:47:36,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 21:47:36,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35414.61 MB 2025-02-14 21:47:36,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 21:47:36,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33137.76 MB 2025-02-14 21:47:36,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:47:36,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:47:36,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:47:36,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:36,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29127.02 MB 2025-02-14 21:47:36,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29894.02 MB 2025-02-14 21:47:36,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:47:36,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35414.61 MB 2025-02-14 21:47:36,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 21:47:36,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:47:36,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30601.81 MB 2025-02-14 21:47:36,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:47:36,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:47:36,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:47:36,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:36,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30306.91 MB 2025-02-14 21:47:36,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30534.46 MB 2025-02-14 21:47:36,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.55 MB 2025-02-14 21:47:36,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35831.94 MB 2025-02-14 21:47:36,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 21:47:36,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:47:36,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30770.95 MB 2025-02-14 21:47:36,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:47:36,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:47:36,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.44 seconds 2025-02-14 21:47:36,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:36,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17762.80 MB 2025-02-14 21:47:36,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30735.16 MB 2025-02-14 21:47:36,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12972.36 MB 2025-02-14 21:47:36,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51030.00 MB 2025-02-14 21:47:36,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 21:47:36,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15198.06 MB 2025-02-14 21:47:36,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30770.95 MB 2025-02-14 21:47:36,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:47:36,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:47:36,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:47:36,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:36,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30735.16 MB 2025-02-14 21:47:36,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22761.48 MB 2025-02-14 21:47:36,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7973.69 MB 2025-02-14 21:47:36,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35831.94 MB 2025-02-14 21:47:36,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 21:47:36,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:47:36,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33242.22 MB 2025-02-14 21:47:36,572 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 21:47:36,573 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:47:36,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:47:36,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:47:36,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:47:36,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:47:36,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22761.48 MB 2025-02-14 21:47:36,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31184.68 MB 2025-02-14 21:47:36,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 21:47:36,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35831.94 MB 2025-02-14 21:47:36,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44207.96 MB 2025-02-14 21:47:36,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 21:47:36,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31184.68 MB 2025-02-14 21:47:36,742 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 21:47:36,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:47:36,744 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:47:36,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:47:36,745 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:47:36,749 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:47:36,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:47:36,751 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:47:36,751 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:48:38,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:48:38,558 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:48:38,563 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:48:38,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:48:38,567 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2463, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:48:38,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:48:38,568 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2463, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:49:16,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:49:16,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:49:16,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.83 seconds 2025-02-14 21:49:16,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:16,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30132.45 MB 2025-02-14 21:49:16,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38848.87 MB 2025-02-14 21:49:16,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8716.42 MB 2025-02-14 21:49:16,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69751.28 MB 2025-02-14 21:49:16,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44000.35 MB 2025-02-14 21:49:16,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25750.93 MB 2025-02-14 21:49:16,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47757.55 MB 2025-02-14 21:49:16,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:49:16,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:49:16,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:49:16,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:16,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38848.87 MB 2025-02-14 21:49:16,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28583.39 MB 2025-02-14 21:49:16,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10265.48 MB 2025-02-14 21:49:16,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44000.35 MB 2025-02-14 21:49:16,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 74725.72 MB 2025-02-14 21:49:16,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30725.37 MB 2025-02-14 21:49:16,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64422.05 MB 2025-02-14 21:49:18,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:49:18,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:49:18,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 21:49:18,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:18,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28583.39 MB 2025-02-14 21:49:18,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29114.23 MB 2025-02-14 21:49:18,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:49:18,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74725.72 MB 2025-02-14 21:49:18,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32338.08 MB 2025-02-14 21:49:18,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42387.64 MB 2025-02-14 21:49:18,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33093.82 MB 2025-02-14 21:49:18,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:49:18,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:49:18,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:49:18,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:18,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29114.23 MB 2025-02-14 21:49:18,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31003.44 MB 2025-02-14 21:49:18,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-14 21:49:18,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32338.08 MB 2025-02-14 21:49:18,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34225.52 MB 2025-02-14 21:49:18,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:49:18,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32420.87 MB 2025-02-14 21:49:18,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:49:18,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:49:18,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:49:18,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:18,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31003.44 MB 2025-02-14 21:49:18,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33245.29 MB 2025-02-14 21:49:18,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:49:18,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34225.52 MB 2025-02-14 21:49:18,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40359.69 MB 2025-02-14 21:49:18,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:49:18,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38789.57 MB 2025-02-14 21:49:18,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:49:18,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:49:18,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:49:18,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:18,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29114.23 MB 2025-02-14 21:49:18,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33245.29 MB 2025-02-14 21:49:18,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-14 21:49:18,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32338.08 MB 2025-02-14 21:49:18,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40359.69 MB 2025-02-14 21:49:18,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 21:49:18,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38789.57 MB 2025-02-14 21:49:18,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:49:18,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:49:18,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:49:18,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:18,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34778.83 MB 2025-02-14 21:49:18,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35545.84 MB 2025-02-14 21:49:18,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:49:18,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40359.69 MB 2025-02-14 21:49:18,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40774.93 MB 2025-02-14 21:49:18,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:49:18,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36253.62 MB 2025-02-14 21:49:18,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:49:18,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:49:18,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:49:18,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:18,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35958.73 MB 2025-02-14 21:49:18,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36187.80 MB 2025-02-14 21:49:18,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.07 MB 2025-02-14 21:49:18,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40774.93 MB 2025-02-14 21:49:18,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40774.93 MB 2025-02-14 21:49:18,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:49:18,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36412.04 MB 2025-02-14 21:49:18,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:49:18,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:49:18,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.37 seconds 2025-02-14 21:49:18,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:18,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21550.58 MB 2025-02-14 21:49:18,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36388.50 MB 2025-02-14 21:49:18,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14837.92 MB 2025-02-14 21:49:18,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61167.63 MB 2025-02-14 21:49:18,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40774.93 MB 2025-02-14 21:49:18,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20392.71 MB 2025-02-14 21:49:18,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36412.04 MB 2025-02-14 21:49:19,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:49:19,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:49:19,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:49:19,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:19,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36388.50 MB 2025-02-14 21:49:19,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26549.26 MB 2025-02-14 21:49:19,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9839.25 MB 2025-02-14 21:49:19,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40774.93 MB 2025-02-14 21:49:19,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40774.93 MB 2025-02-14 21:49:19,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:49:19,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38895.56 MB 2025-02-14 21:49:19,225 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 21:49:19,225 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:49:19,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:49:19,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:49:19,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:49:19,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:19,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26549.26 MB 2025-02-14 21:49:19,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34972.46 MB 2025-02-14 21:49:19,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 21:49:19,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40774.93 MB 2025-02-14 21:49:19,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44962.94 MB 2025-02-14 21:49:19,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 21:49:19,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34972.46 MB 2025-02-14 21:49:19,388 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 21:49:19,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:49:19,389 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:49:19,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:49:19,390 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:49:19,395 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:49:19,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:49:19,396 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:49:19,396 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:49:36,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:49:36,377 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:49:36,381 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:49:36,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:49:36,385 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1413, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:49:36,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:49:36,386 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1413, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:49:58,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:49:58,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:49:58,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.99 seconds 2025-02-14 21:49:58,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:58,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22814.72 MB 2025-02-14 21:49:58,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27815.24 MB 2025-02-14 21:49:58,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5000.53 MB 2025-02-14 21:49:58,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53338.96 MB 2025-02-14 21:49:58,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35678.85 MB 2025-02-14 21:49:58,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17660.12 MB 2025-02-14 21:49:58,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36815.94 MB 2025-02-14 21:49:58,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:49:58,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:49:58,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:49:58,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:49:58,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27815.24 MB 2025-02-14 21:49:58,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23123.60 MB 2025-02-14 21:49:58,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4691.65 MB 2025-02-14 21:49:58,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35678.85 MB 2025-02-14 21:49:58,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47968.16 MB 2025-02-14 21:49:58,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12289.31 MB 2025-02-14 21:49:58,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42716.98 MB 2025-02-14 21:50:00,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:50:00,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:50:00,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:50:00,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:00,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23123.60 MB 2025-02-14 21:50:00,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23654.44 MB 2025-02-14 21:50:00,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:50:00,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47968.16 MB 2025-02-14 21:50:00,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-14 21:50:00,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17291.02 MB 2025-02-14 21:50:00,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27632.99 MB 2025-02-14 21:50:00,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:50:00,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:50:00,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:50:00,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:00,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.44 MB 2025-02-14 21:50:00,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25543.97 MB 2025-02-14 21:50:00,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:50:00,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 21:50:00,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-14 21:50:00,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:50:00,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26961.40 MB 2025-02-14 21:50:00,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:50:00,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:50:00,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:50:00,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:00,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25543.97 MB 2025-02-14 21:50:00,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27785.83 MB 2025-02-14 21:50:00,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:50:00,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 21:50:00,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35867.59 MB 2025-02-14 21:50:00,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:50:00,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33330.11 MB 2025-02-14 21:50:00,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:50:00,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:50:00,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:50:00,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:00,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.44 MB 2025-02-14 21:50:00,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27785.83 MB 2025-02-14 21:50:00,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:50:00,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 21:50:00,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35867.59 MB 2025-02-14 21:50:00,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 21:50:00,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33330.11 MB 2025-02-14 21:50:00,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:50:00,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:50:00,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:50:00,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:00,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29319.37 MB 2025-02-14 21:50:00,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30086.37 MB 2025-02-14 21:50:00,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:50:00,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35867.59 MB 2025-02-14 21:50:00,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36284.92 MB 2025-02-14 21:50:00,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:50:00,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30794.16 MB 2025-02-14 21:50:00,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:50:00,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:50:00,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:50:00,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:00,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30499.26 MB 2025-02-14 21:50:00,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30728.80 MB 2025-02-14 21:50:00,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.54 MB 2025-02-14 21:50:00,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36284.92 MB 2025-02-14 21:50:00,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36284.92 MB 2025-02-14 21:50:00,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:50:00,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30967.92 MB 2025-02-14 21:50:00,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:50:00,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:50:00,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.43 seconds 2025-02-14 21:50:00,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:00,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17891.71 MB 2025-02-14 21:50:00,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30929.58 MB 2025-02-14 21:50:00,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13037.87 MB 2025-02-14 21:50:00,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53338.96 MB 2025-02-14 21:50:00,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36284.92 MB 2025-02-14 21:50:00,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17054.04 MB 2025-02-14 21:50:00,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30967.92 MB 2025-02-14 21:50:01,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:50:01,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:50:01,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:50:01,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:01,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30929.58 MB 2025-02-14 21:50:01,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22891.53 MB 2025-02-14 21:50:01,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8038.05 MB 2025-02-14 21:50:01,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36284.92 MB 2025-02-14 21:50:01,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36284.92 MB 2025-02-14 21:50:01,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:50:01,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33437.56 MB 2025-02-14 21:50:01,104 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 21:50:01,105 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 21:50:01,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:50:01,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:50:01,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:50:01,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:01,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22891.53 MB 2025-02-14 21:50:01,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31318.03 MB 2025-02-14 21:50:01,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 21:50:01,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36284.92 MB 2025-02-14 21:50:01,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44663.05 MB 2025-02-14 21:50:01,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 21:50:01,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31318.03 MB 2025-02-14 21:50:01,267 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 21:50:01,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:01,268 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:50:01,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:01,269 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:50:01,274 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:50:01,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:01,275 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:50:01,275 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 21:50:10,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:10,647 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:50:10,652 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:50:10,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:10,657 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 349, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:50:10,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:10,658 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 349, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:50:16,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:50:16,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:50:16,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.47 seconds 2025-02-14 21:50:16,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:16,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15400.59 MB 2025-02-14 21:50:16,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16635.69 MB 2025-02-14 21:50:16,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1235.09 MB 2025-02-14 21:50:16,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57229.18 MB 2025-02-14 21:50:16,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25165.82 MB 2025-02-14 21:50:16,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32063.36 MB 2025-02-14 21:50:16,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25551.44 MB 2025-02-14 21:50:16,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:50:16,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:50:16,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:50:16,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:16,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16635.69 MB 2025-02-14 21:50:16,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17142.72 MB 2025-02-14 21:50:16,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 507.03 MB 2025-02-14 21:50:16,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25165.82 MB 2025-02-14 21:50:16,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25165.82 MB 2025-02-14 21:50:16,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:50:16,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21401.16 MB 2025-02-14 21:50:17,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:50:17,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:50:17,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.62 seconds 2025-02-14 21:50:17,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:17,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17142.72 MB 2025-02-14 21:50:17,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17588.63 MB 2025-02-14 21:50:17,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 445.91 MB 2025-02-14 21:50:17,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25165.82 MB 2025-02-14 21:50:17,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24222.11 MB 2025-02-14 21:50:17,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 21:50:17,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21567.17 MB 2025-02-14 21:50:17,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:50:17,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:50:17,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:50:17,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:17,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17588.63 MB 2025-02-14 21:50:17,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19176.17 MB 2025-02-14 21:50:17,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1587.54 MB 2025-02-14 21:50:17,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24222.11 MB 2025-02-14 21:50:17,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24222.11 MB 2025-02-14 21:50:17,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:50:17,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20366.81 MB 2025-02-14 21:50:17,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:50:17,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:50:17,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 21:50:17,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:17,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19176.17 MB 2025-02-14 21:50:17,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21059.34 MB 2025-02-14 21:50:17,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1883.17 MB 2025-02-14 21:50:17,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24222.11 MB 2025-02-14 21:50:17,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 21:50:17,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3963.62 MB 2025-02-14 21:50:17,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25716.53 MB 2025-02-14 21:50:17,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:50:17,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:50:17,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 21:50:17,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:17,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17588.63 MB 2025-02-14 21:50:17,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21059.34 MB 2025-02-14 21:50:17,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3470.71 MB 2025-02-14 21:50:17,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24222.11 MB 2025-02-14 21:50:17,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28185.72 MB 2025-02-14 21:50:17,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3963.62 MB 2025-02-14 21:50:17,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25716.53 MB 2025-02-14 21:50:18,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:50:18,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:50:18,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 21:50:18,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:18,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22347.52 MB 2025-02-14 21:50:18,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22991.80 MB 2025-02-14 21:50:18,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.28 MB 2025-02-14 21:50:18,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28185.72 MB 2025-02-14 21:50:18,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28533.85 MB 2025-02-14 21:50:18,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 348.13 MB 2025-02-14 21:50:18,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23586.34 MB 2025-02-14 21:50:18,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:50:18,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:50:18,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:50:18,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:18,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23338.63 MB 2025-02-14 21:50:18,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23566.71 MB 2025-02-14 21:50:18,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 21:50:18,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28533.85 MB 2025-02-14 21:50:18,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28533.85 MB 2025-02-14 21:50:18,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:50:18,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23717.34 MB 2025-02-14 21:50:18,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:50:18,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:50:18,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.47 seconds 2025-02-14 21:50:18,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:18,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14184.65 MB 2025-02-14 21:50:18,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23767.78 MB 2025-02-14 21:50:18,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9583.13 MB 2025-02-14 21:50:18,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57229.18 MB 2025-02-14 21:50:18,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28533.85 MB 2025-02-14 21:50:18,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28695.33 MB 2025-02-14 21:50:18,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23767.78 MB 2025-02-14 21:50:18,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:50:18,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:50:18,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:50:18,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:18,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23767.78 MB 2025-02-14 21:50:18,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26781.81 MB 2025-02-14 21:50:18,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 21:50:18,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28533.85 MB 2025-02-14 21:50:18,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28533.85 MB 2025-02-14 21:50:18,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:50:18,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27083.18 MB 2025-02-14 21:50:18,422 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:50:18,423 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1,'] 2025-02-14 21:50:18,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:50:18,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:50:18,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:50:18,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:50:18,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18887.00 MB 2025-02-14 21:50:18,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27326.03 MB 2025-02-14 21:50:18,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:50:18,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28533.85 MB 2025-02-14 21:50:18,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36924.56 MB 2025-02-14 21:50:18,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:50:18,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27326.03 MB 2025-02-14 21:50:18,591 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:50:18,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:18,593 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:50:18,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:18,594 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:50:18,598 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:50:18,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:50:18,599 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:50:18,599 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1,'] 2025-02-14 21:51:08,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:51:08,876 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:51:08,881 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:51:08,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:51:08,886 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:51:08,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:51:08,887 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:51:11,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:51:11,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:51:11,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.68 seconds 2025-02-14 21:51:11,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:11,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-14 21:51:11,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-14 21:51:11,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-14 21:51:11,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49509.56 MB 2025-02-14 21:51:11,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22206.74 MB 2025-02-14 21:51:11,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27302.82 MB 2025-02-14 21:51:11,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23652.54 MB 2025-02-14 21:51:11,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:51:11,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:51:11,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:51:11,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:11,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-14 21:51:11,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15003.98 MB 2025-02-14 21:51:11,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.04 MB 2025-02-14 21:51:11,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22206.74 MB 2025-02-14 21:51:11,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22206.74 MB 2025-02-14 21:51:11,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:51:11,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17107.97 MB 2025-02-14 21:51:12,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:51:12,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:51:12,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 21:51:12,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15003.98 MB 2025-02-14 21:51:12,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15217.65 MB 2025-02-14 21:51:12,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 21:51:12,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22206.74 MB 2025-02-14 21:51:12,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-14 21:51:12,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -396.36 MB 2025-02-14 21:51:12,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19173.64 MB 2025-02-14 21:51:12,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:51:12,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:51:12,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 21:51:12,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-14 21:51:12,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15977.94 MB 2025-02-14 21:51:12,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 21:51:12,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-14 21:51:12,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-14 21:51:12,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:51:12,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16548.46 MB 2025-02-14 21:51:12,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:51:12,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:51:12,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:51:12,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.94 MB 2025-02-14 21:51:12,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.52 MB 2025-02-14 21:51:12,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.59 MB 2025-02-14 21:51:12,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-14 21:51:12,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-14 21:51:12,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:51:12,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19112.06 MB 2025-02-14 21:51:12,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:51:12,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:51:12,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 21:51:12,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15217.58 MB 2025-02-14 21:51:12,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16880.52 MB 2025-02-14 21:51:12,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.94 MB 2025-02-14 21:51:12,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-14 21:51:12,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21810.38 MB 2025-02-14 21:51:12,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:51:12,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19112.06 MB 2025-02-14 21:51:12,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:51:12,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:51:12,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:51:12,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17497.77 MB 2025-02-14 21:51:12,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17806.49 MB 2025-02-14 21:51:12,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 21:51:12,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21810.38 MB 2025-02-14 21:51:12,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21973.96 MB 2025-02-14 21:51:12,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 21:51:12,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18100.63 MB 2025-02-14 21:51:12,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:51:12,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:51:12,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:51:12,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17972.69 MB 2025-02-14 21:51:12,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18200.61 MB 2025-02-14 21:51:12,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.92 MB 2025-02-14 21:51:12,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21973.96 MB 2025-02-14 21:51:12,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21973.96 MB 2025-02-14 21:51:12,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:51:12,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18223.17 MB 2025-02-14 21:51:12,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:51:12,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:51:12,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.65 seconds 2025-02-14 21:51:12,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-14 21:51:12,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18401.68 MB 2025-02-14 21:51:12,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4826.75 MB 2025-02-14 21:51:12,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49509.56 MB 2025-02-14 21:51:12,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21973.96 MB 2025-02-14 21:51:12,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27535.61 MB 2025-02-14 21:51:12,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18401.68 MB 2025-02-14 21:51:12,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:51:12,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:51:12,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:51:12,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18401.68 MB 2025-02-14 21:51:12,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17451.41 MB 2025-02-14 21:51:12,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -950.27 MB 2025-02-14 21:51:12,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21973.96 MB 2025-02-14 21:51:12,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21973.96 MB 2025-02-14 21:51:12,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:51:12,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19205.42 MB 2025-02-14 21:51:12,823 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:51:12,824 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 1,'] 2025-02-14 21:51:12,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:51:12,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:51:12,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:51:12,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:51:12,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17451.41 MB 2025-02-14 21:51:12,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25890.44 MB 2025-02-14 21:51:12,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:51:12,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21973.96 MB 2025-02-14 21:51:12,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30364.66 MB 2025-02-14 21:51:12,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:51:12,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25890.44 MB 2025-02-14 21:51:12,989 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:51:12,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:51:12,990 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:51:12,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:51:12,991 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:51:12,996 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:51:12,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:51:12,997 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:51:12,997 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 1,'] 2025-02-14 21:52:16,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:52:16,397 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:52:16,402 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:52:16,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:52:16,406 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:52:16,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:52:16,407 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:52:35,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:52:35,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:52:35,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.22 seconds 2025-02-14 21:52:35,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:35,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21727.68 MB 2025-02-14 21:52:35,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26176.14 MB 2025-02-14 21:52:35,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4448.45 MB 2025-02-14 21:52:35,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42949.67 MB 2025-02-14 21:52:35,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33306.97 MB 2025-02-14 21:52:35,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9642.70 MB 2025-02-14 21:52:35,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35049.43 MB 2025-02-14 21:52:35,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:52:35,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:52:35,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 21:52:35,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:35,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26176.14 MB 2025-02-14 21:52:35,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22312.60 MB 2025-02-14 21:52:35,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3863.53 MB 2025-02-14 21:52:35,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33306.97 MB 2025-02-14 21:52:35,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45369.79 MB 2025-02-14 21:52:35,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12062.82 MB 2025-02-14 21:52:35,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39347.90 MB 2025-02-14 21:52:37,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:52:37,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:52:37,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 21:52:37,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:37,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22312.60 MB 2025-02-14 21:52:37,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22843.44 MB 2025-02-14 21:52:37,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:52:37,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45369.79 MB 2025-02-14 21:52:37,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-14 21:52:37,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17452.50 MB 2025-02-14 21:52:37,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26821.99 MB 2025-02-14 21:52:37,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:52:37,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:52:37,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:52:37,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:37,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 21:52:37,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24732.98 MB 2025-02-14 21:52:37,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:52:37,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 21:52:37,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-14 21:52:37,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:52:37,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26150.41 MB 2025-02-14 21:52:37,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:52:37,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:52:37,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 21:52:37,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:37,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24732.98 MB 2025-02-14 21:52:37,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 21:52:37,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:52:37,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 21:52:37,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34523.32 MB 2025-02-14 21:52:37,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 21:52:37,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 21:52:37,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:52:37,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:52:37,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 21:52:37,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:37,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 21:52:37,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 21:52:37,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:52:37,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 21:52:37,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34523.32 MB 2025-02-14 21:52:37,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 21:52:37,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 21:52:38,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:52:38,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:52:38,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:52:38,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:38,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28508.38 MB 2025-02-14 21:52:38,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.38 MB 2025-02-14 21:52:38,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:52:38,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34523.32 MB 2025-02-14 21:52:38,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34940.65 MB 2025-02-14 21:52:38,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:52:38,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29983.17 MB 2025-02-14 21:52:38,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:52:38,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:52:38,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:52:38,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:38,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.27 MB 2025-02-14 21:52:38,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29917.82 MB 2025-02-14 21:52:38,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.55 MB 2025-02-14 21:52:38,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34940.65 MB 2025-02-14 21:52:38,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34940.65 MB 2025-02-14 21:52:38,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:52:38,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30161.01 MB 2025-02-14 21:52:38,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:52:38,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:52:38,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.64 seconds 2025-02-14 21:52:38,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:38,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17348.19 MB 2025-02-14 21:52:38,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30118.38 MB 2025-02-14 21:52:38,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12770.18 MB 2025-02-14 21:52:38,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42949.67 MB 2025-02-14 21:52:38,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34940.65 MB 2025-02-14 21:52:38,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8009.02 MB 2025-02-14 21:52:38,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30161.01 MB 2025-02-14 21:52:38,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:52:38,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:52:38,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:52:38,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:38,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30118.38 MB 2025-02-14 21:52:38,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22344.58 MB 2025-02-14 21:52:38,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7773.79 MB 2025-02-14 21:52:38,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34940.65 MB 2025-02-14 21:52:38,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34940.65 MB 2025-02-14 21:52:38,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:52:38,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32623.59 MB 2025-02-14 21:52:38,336 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 21:52:38,336 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 21:52:38,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:52:38,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:52:38,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:52:38,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:52:38,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.58 MB 2025-02-14 21:52:38,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30762.33 MB 2025-02-14 21:52:38,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-14 21:52:38,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34940.65 MB 2025-02-14 21:52:38,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43308.29 MB 2025-02-14 21:52:38,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 21:52:38,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30762.33 MB 2025-02-14 21:52:38,498 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 21:52:38,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:52:38,500 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:52:38,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:52:38,501 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:52:38,505 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:52:38,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:52:38,506 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:52:38,506 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 21:54:39,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:54:39,341 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:54:39,346 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:54:39,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:54:39,350 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1838, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:54:39,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:54:39,351 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1838, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:55:07,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:55:07,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:55:07,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.12 seconds 2025-02-14 21:55:07,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:07,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25776.18 MB 2025-02-14 21:55:07,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32281.48 MB 2025-02-14 21:55:07,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6505.30 MB 2025-02-14 21:55:07,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51675.92 MB 2025-02-14 21:55:07,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37169.92 MB 2025-02-14 21:55:07,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14506.00 MB 2025-02-14 21:55:07,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41137.10 MB 2025-02-14 21:55:07,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:55:07,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:55:07,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 21:55:07,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:07,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32281.48 MB 2025-02-14 21:55:07,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25333.04 MB 2025-02-14 21:55:07,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6948.45 MB 2025-02-14 21:55:07,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37169.92 MB 2025-02-14 21:55:07,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56897.83 MB 2025-02-14 21:55:07,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19727.91 MB 2025-02-14 21:55:07,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48447.21 MB 2025-02-14 21:55:09,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:55:09,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:55:09,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:55:09,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:09,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25333.04 MB 2025-02-14 21:55:09,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25863.88 MB 2025-02-14 21:55:09,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:55:09,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56897.83 MB 2025-02-14 21:55:09,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 21:55:09,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24817.70 MB 2025-02-14 21:55:09,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29842.43 MB 2025-02-14 21:55:09,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:55:09,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:55:09,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:55:09,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:09,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25863.88 MB 2025-02-14 21:55:09,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27753.41 MB 2025-02-14 21:55:09,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:55:09,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 21:55:09,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 21:55:09,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:55:09,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29170.84 MB 2025-02-14 21:55:09,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:55:09,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:55:09,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:55:09,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:09,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27753.41 MB 2025-02-14 21:55:09,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29995.27 MB 2025-02-14 21:55:09,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:55:09,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 21:55:09,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37742.44 MB 2025-02-14 21:55:09,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:55:09,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35539.55 MB 2025-02-14 21:55:09,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:55:09,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:55:09,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:55:09,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:09,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25863.88 MB 2025-02-14 21:55:09,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29995.27 MB 2025-02-14 21:55:09,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:55:09,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 21:55:09,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37742.44 MB 2025-02-14 21:55:09,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:55:09,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35539.55 MB 2025-02-14 21:55:09,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:55:09,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:55:09,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:55:09,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:09,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31528.81 MB 2025-02-14 21:55:09,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32295.81 MB 2025-02-14 21:55:09,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:55:09,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37742.44 MB 2025-02-14 21:55:09,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 21:55:09,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 21:55:09,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33003.60 MB 2025-02-14 21:55:09,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:55:09,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:55:09,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:55:09,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:09,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32708.70 MB 2025-02-14 21:55:09,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32936.21 MB 2025-02-14 21:55:09,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.51 MB 2025-02-14 21:55:09,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38159.78 MB 2025-02-14 21:55:09,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 21:55:09,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:55:09,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33179.59 MB 2025-02-14 21:55:09,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:55:09,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:55:09,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.59 seconds 2025-02-14 21:55:09,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:09,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19372.45 MB 2025-02-14 21:55:09,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33136.92 MB 2025-02-14 21:55:09,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13764.47 MB 2025-02-14 21:55:09,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51675.92 MB 2025-02-14 21:55:09,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 21:55:09,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13516.14 MB 2025-02-14 21:55:09,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33179.59 MB 2025-02-14 21:55:10,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:55:10,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:55:10,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:55:10,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:10,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21362.62 MB 2025-02-14 21:55:10,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24371.12 MB 2025-02-14 21:55:10,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-14 21:55:10,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38159.78 MB 2025-02-14 21:55:10,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 21:55:10,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:55:10,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24671.93 MB 2025-02-14 21:55:10,227 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 21:55:10,227 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:55:10,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:55:10,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:55:10,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:55:10,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:55:10,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24371.12 MB 2025-02-14 21:55:10,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32794.33 MB 2025-02-14 21:55:10,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 21:55:10,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38159.78 MB 2025-02-14 21:55:10,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46535.80 MB 2025-02-14 21:55:10,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 21:55:10,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32794.33 MB 2025-02-14 21:55:10,398 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 21:55:10,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:55:10,400 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:55:10,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:55:10,400 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:55:10,405 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:55:10,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:55:10,406 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:55:10,406 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:55:56,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:55:56,896 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:55:56,901 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:55:56,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:55:56,904 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2771, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:55:56,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:55:56,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2771, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:56:39,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:56:39,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:56:39,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.77 seconds 2025-02-14 21:56:39,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:39,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32279.41 MB 2025-02-14 21:56:39,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42085.83 MB 2025-02-14 21:56:39,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9806.41 MB 2025-02-14 21:56:39,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74226.60 MB 2025-02-14 21:56:39,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47238.35 MB 2025-02-14 21:56:39,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26988.25 MB 2025-02-14 21:56:39,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51892.24 MB 2025-02-14 21:56:39,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:56:39,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:56:39,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 21:56:39,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:39,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42085.83 MB 2025-02-14 21:56:39,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30185.35 MB 2025-02-14 21:56:39,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11900.48 MB 2025-02-14 21:56:39,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47238.35 MB 2025-02-14 21:56:39,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 81057.02 MB 2025-02-14 21:56:39,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33818.67 MB 2025-02-14 21:56:39,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69941.67 MB 2025-02-14 21:56:41,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:56:41,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:56:41,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 21:56:41,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:41,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30185.35 MB 2025-02-14 21:56:41,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30716.19 MB 2025-02-14 21:56:41,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:56:41,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81057.02 MB 2025-02-14 21:56:41,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33940.31 MB 2025-02-14 21:56:41,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -47116.71 MB 2025-02-14 21:56:41,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34695.78 MB 2025-02-14 21:56:41,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:56:41,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:56:41,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:56:41,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:41,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30716.19 MB 2025-02-14 21:56:41,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32605.66 MB 2025-02-14 21:56:41,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.47 MB 2025-02-14 21:56:41,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33940.31 MB 2025-02-14 21:56:41,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35827.74 MB 2025-02-14 21:56:41,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:56:41,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34023.09 MB 2025-02-14 21:56:42,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:56:42,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:56:42,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 21:56:42,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:42,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32605.66 MB 2025-02-14 21:56:42,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34847.52 MB 2025-02-14 21:56:42,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:56:42,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35827.74 MB 2025-02-14 21:56:42,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41961.91 MB 2025-02-14 21:56:42,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:56:42,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40391.80 MB 2025-02-14 21:56:42,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:56:42,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:56:42,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:56:42,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:42,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30716.19 MB 2025-02-14 21:56:42,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34847.52 MB 2025-02-14 21:56:42,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-14 21:56:42,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33940.31 MB 2025-02-14 21:56:42,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41961.91 MB 2025-02-14 21:56:42,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 21:56:42,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40391.80 MB 2025-02-14 21:56:42,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:56:42,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:56:42,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:56:42,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:42,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36381.06 MB 2025-02-14 21:56:42,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37148.06 MB 2025-02-14 21:56:42,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:56:42,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41961.91 MB 2025-02-14 21:56:42,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42377.15 MB 2025-02-14 21:56:42,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:56:42,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37855.85 MB 2025-02-14 21:56:42,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:56:42,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:56:42,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:56:42,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:42,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37560.95 MB 2025-02-14 21:56:42,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37789.91 MB 2025-02-14 21:56:42,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 21:56:42,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42377.15 MB 2025-02-14 21:56:42,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42377.15 MB 2025-02-14 21:56:42,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:56:42,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38011.85 MB 2025-02-14 21:56:42,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:56:42,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:56:42,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.34 seconds 2025-02-14 21:56:42,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:42,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22624.06 MB 2025-02-14 21:56:42,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37990.71 MB 2025-02-14 21:56:42,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15366.65 MB 2025-02-14 21:56:42,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64569.21 MB 2025-02-14 21:56:42,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42377.15 MB 2025-02-14 21:56:42,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22192.06 MB 2025-02-14 21:56:42,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38011.85 MB 2025-02-14 21:56:42,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:56:42,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:56:42,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:56:42,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:42,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37990.71 MB 2025-02-14 21:56:42,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27624.26 MB 2025-02-14 21:56:42,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10366.45 MB 2025-02-14 21:56:42,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42377.15 MB 2025-02-14 21:56:42,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42377.15 MB 2025-02-14 21:56:42,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:56:42,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40499.00 MB 2025-02-14 21:56:42,538 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 21:56:42,538 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:56:42,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:56:42,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:56:42,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:56:42,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:56:42,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27624.26 MB 2025-02-14 21:56:42,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36051.59 MB 2025-02-14 21:56:42,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 21:56:42,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42377.15 MB 2025-02-14 21:56:42,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46567.26 MB 2025-02-14 21:56:42,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 21:56:42,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36051.59 MB 2025-02-14 21:56:42,703 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 21:56:42,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:56:42,704 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:56:42,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:56:42,705 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:56:42,710 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:56:42,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:56:42,711 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:56:42,711 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:57:31,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:57:31,658 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:57:31,663 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:57:31,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:57:31,666 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1035, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:57:31,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:57:31,667 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1035, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:57:47,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:57:47,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:57:47,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.04 seconds 2025-02-14 21:57:47,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:47,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20180.75 MB 2025-02-14 21:57:47,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23844.48 MB 2025-02-14 21:57:47,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3663.72 MB 2025-02-14 21:57:47,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54947.48 MB 2025-02-14 21:57:47,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25971.13 MB 2025-02-14 21:57:47,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28976.35 MB 2025-02-14 21:57:47,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32823.82 MB 2025-02-14 21:57:47,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:57:47,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:57:47,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:57:47,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:47,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23844.48 MB 2025-02-14 21:57:47,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21159.54 MB 2025-02-14 21:57:47,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2684.94 MB 2025-02-14 21:57:47,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25971.13 MB 2025-02-14 21:57:47,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42586.87 MB 2025-02-14 21:57:47,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16615.74 MB 2025-02-14 21:57:47,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34844.32 MB 2025-02-14 21:57:49,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:57:49,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:57:49,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 21:57:49,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:49,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21159.54 MB 2025-02-14 21:57:49,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21690.38 MB 2025-02-14 21:57:49,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:57:49,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42586.87 MB 2025-02-14 21:57:49,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25140.66 MB 2025-02-14 21:57:49,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17446.21 MB 2025-02-14 21:57:49,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25669.97 MB 2025-02-14 21:57:49,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:57:49,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:57:49,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:57:49,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:49,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21690.38 MB 2025-02-14 21:57:49,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23579.92 MB 2025-02-14 21:57:49,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:57:49,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25140.66 MB 2025-02-14 21:57:49,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27028.09 MB 2025-02-14 21:57:49,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 21:57:49,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24997.35 MB 2025-02-14 21:57:49,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:57:49,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:57:49,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:57:49,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:49,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23579.92 MB 2025-02-14 21:57:49,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25821.77 MB 2025-02-14 21:57:49,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:57:49,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27028.09 MB 2025-02-14 21:57:49,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33162.26 MB 2025-02-14 21:57:49,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 21:57:49,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31366.05 MB 2025-02-14 21:57:49,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:57:49,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:57:49,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:57:49,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:49,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21690.38 MB 2025-02-14 21:57:49,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25821.77 MB 2025-02-14 21:57:49,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:57:49,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25140.66 MB 2025-02-14 21:57:49,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33162.26 MB 2025-02-14 21:57:49,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 21:57:49,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31366.05 MB 2025-02-14 21:57:50,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:57:50,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:57:50,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:57:50,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:50,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27355.32 MB 2025-02-14 21:57:50,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28122.32 MB 2025-02-14 21:57:50,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:57:50,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33162.26 MB 2025-02-14 21:57:50,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33575.40 MB 2025-02-14 21:57:50,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 21:57:50,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28830.11 MB 2025-02-14 21:57:50,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:57:50,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:57:50,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:57:50,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:50,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28535.21 MB 2025-02-14 21:57:50,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28765.58 MB 2025-02-14 21:57:50,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.37 MB 2025-02-14 21:57:50,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33575.40 MB 2025-02-14 21:57:50,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33575.40 MB 2025-02-14 21:57:50,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:57:50,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28974.47 MB 2025-02-14 21:57:50,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:57:50,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:57:50,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.45 seconds 2025-02-14 21:57:50,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:50,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16574.73 MB 2025-02-14 21:57:50,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28966.65 MB 2025-02-14 21:57:50,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12391.92 MB 2025-02-14 21:57:50,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54947.48 MB 2025-02-14 21:57:50,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33575.40 MB 2025-02-14 21:57:50,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21372.08 MB 2025-02-14 21:57:50,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28974.47 MB 2025-02-14 21:57:50,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:57:50,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:57:50,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:57:50,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:50,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28966.65 MB 2025-02-14 21:57:50,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21579.12 MB 2025-02-14 21:57:50,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7387.53 MB 2025-02-14 21:57:50,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33575.40 MB 2025-02-14 21:57:50,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33575.40 MB 2025-02-14 21:57:50,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:57:50,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31478.32 MB 2025-02-14 21:57:50,408 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 21:57:50,408 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:57:50,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:57:50,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:57:50,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:57:50,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:57:50,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21579.12 MB 2025-02-14 21:57:50,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30018.14 MB 2025-02-14 21:57:50,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 21:57:50,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33575.40 MB 2025-02-14 21:57:50,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41966.11 MB 2025-02-14 21:57:50,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 21:57:50,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30018.14 MB 2025-02-14 21:57:50,570 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 21:57:50,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:57:50,571 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:57:50,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:57:50,572 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:57:50,577 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:57:50,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:57:50,578 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:57:50,578 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 21:58:04,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:58:04,096 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:58:04,101 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:58:04,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:58:04,104 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1132, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:58:04,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:58:04,105 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1132, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:58:21,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:58:21,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:58:21,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.54 seconds 2025-02-14 21:58:21,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:21,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20856.66 MB 2025-02-14 21:58:21,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24862.75 MB 2025-02-14 21:58:21,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4006.08 MB 2025-02-14 21:58:21,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54551.12 MB 2025-02-14 21:58:21,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30507.27 MB 2025-02-14 21:58:21,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24043.85 MB 2025-02-14 21:58:21,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33725.42 MB 2025-02-14 21:58:21,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:58:21,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:58:21,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 21:58:21,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:21,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24862.75 MB 2025-02-14 21:58:21,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21663.81 MB 2025-02-14 21:58:21,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3198.93 MB 2025-02-14 21:58:21,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30507.27 MB 2025-02-14 21:58:21,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43224.40 MB 2025-02-14 21:58:21,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12717.13 MB 2025-02-14 21:58:21,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37082.14 MB 2025-02-14 21:58:23,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:58:23,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:58:23,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 21:58:23,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:23,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21663.81 MB 2025-02-14 21:58:23,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22194.66 MB 2025-02-14 21:58:23,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 21:58:23,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43224.40 MB 2025-02-14 21:58:23,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28624.03 MB 2025-02-14 21:58:23,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14600.37 MB 2025-02-14 21:58:23,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26173.20 MB 2025-02-14 21:58:23,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:58:23,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:58:23,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:58:23,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:23,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.66 MB 2025-02-14 21:58:23,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24084.19 MB 2025-02-14 21:58:23,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 21:58:23,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28624.03 MB 2025-02-14 21:58:23,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28624.03 MB 2025-02-14 21:58:23,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:58:23,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25501.62 MB 2025-02-14 21:58:23,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:58:23,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:58:23,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 21:58:23,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:23,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24084.19 MB 2025-02-14 21:58:23,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26326.05 MB 2025-02-14 21:58:23,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 21:58:23,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28624.03 MB 2025-02-14 21:58:23,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-14 21:58:23,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:58:23,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31870.33 MB 2025-02-14 21:58:23,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:58:23,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:58:23,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 21:58:23,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:23,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.66 MB 2025-02-14 21:58:23,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26326.05 MB 2025-02-14 21:58:23,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 21:58:23,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28624.03 MB 2025-02-14 21:58:23,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34286.34 MB 2025-02-14 21:58:23,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 21:58:23,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31870.33 MB 2025-02-14 21:58:24,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:58:24,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:58:24,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 21:58:24,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:24,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27859.59 MB 2025-02-14 21:58:24,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28626.59 MB 2025-02-14 21:58:24,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 21:58:24,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34286.34 MB 2025-02-14 21:58:24,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-14 21:58:24,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 21:58:24,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29334.38 MB 2025-02-14 21:58:24,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:58:24,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:58:24,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:58:24,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:24,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29039.48 MB 2025-02-14 21:58:24,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29267.67 MB 2025-02-14 21:58:24,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.19 MB 2025-02-14 21:58:24,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-14 21:58:24,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-14 21:58:24,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:58:24,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29501.03 MB 2025-02-14 21:58:24,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:58:24,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:58:24,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.95 seconds 2025-02-14 21:58:24,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:24,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16912.69 MB 2025-02-14 21:58:24,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29468.15 MB 2025-02-14 21:58:24,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12555.46 MB 2025-02-14 21:58:24,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54551.12 MB 2025-02-14 21:58:24,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-14 21:58:24,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19849.54 MB 2025-02-14 21:58:24,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29501.03 MB 2025-02-14 21:58:24,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:58:24,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:58:24,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 21:58:24,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:24,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29468.15 MB 2025-02-14 21:58:24,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21907.58 MB 2025-02-14 21:58:24,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7560.57 MB 2025-02-14 21:58:24,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-14 21:58:24,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34701.57 MB 2025-02-14 21:58:24,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:58:24,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31972.14 MB 2025-02-14 21:58:24,346 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 21:58:24,346 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 21:58:24,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:58:24,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:58:24,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:58:24,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:58:24,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21907.58 MB 2025-02-14 21:58:24,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30320.52 MB 2025-02-14 21:58:24,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.95 MB 2025-02-14 21:58:24,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-14 21:58:24,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38883.30 MB 2025-02-14 21:58:24,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 21:58:24,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30320.52 MB 2025-02-14 21:58:24,507 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 21:58:24,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:58:24,509 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:58:24,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:58:24,510 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:58:24,514 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:58:24,515 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:58:24,515 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:58:24,516 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 21:59:26,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:59:26,448 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 21:59:26,453 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 21:59:26,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:59:26,457 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 222, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 21:59:26,458 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:59:26,458 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 222, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 21:59:29,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 21:59:29,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 21:59:29,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.43 seconds 2025-02-14 21:59:29,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:29,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14515.64 MB 2025-02-14 21:59:29,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15301.28 MB 2025-02-14 21:59:29,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 785.65 MB 2025-02-14 21:59:29,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47246.74 MB 2025-02-14 21:59:29,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-14 21:59:29,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27478.98 MB 2025-02-14 21:59:29,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24213.50 MB 2025-02-14 21:59:29,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 21:59:29,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 21:59:29,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:59:29,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:29,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15301.28 MB 2025-02-14 21:59:29,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15682.65 MB 2025-02-14 21:59:29,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 381.36 MB 2025-02-14 21:59:29,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-14 21:59:29,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20554.19 MB 2025-02-14 21:59:29,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 786.43 MB 2025-02-14 21:59:29,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18469.84 MB 2025-02-14 21:59:30,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 21:59:30,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 21:59:30,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.05 seconds 2025-02-14 21:59:30,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:30,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15682.65 MB 2025-02-14 21:59:30,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15977.27 MB 2025-02-14 21:59:30,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-14 21:59:30,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20554.19 MB 2025-02-14 21:59:30,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20554.19 MB 2025-02-14 21:59:30,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:59:30,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19937.23 MB 2025-02-14 21:59:30,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 21:59:30,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 21:59:30,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:59:30,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:30,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.27 MB 2025-02-14 21:59:30,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17025.70 MB 2025-02-14 21:59:30,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.44 MB 2025-02-14 21:59:30,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20554.19 MB 2025-02-14 21:59:30,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20554.19 MB 2025-02-14 21:59:30,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:59:30,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17812.38 MB 2025-02-14 21:59:31,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 21:59:31,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 21:59:31,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 21:59:31,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:31,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17025.70 MB 2025-02-14 21:59:31,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18269.96 MB 2025-02-14 21:59:31,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1244.26 MB 2025-02-14 21:59:31,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20554.19 MB 2025-02-14 21:59:31,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22913.48 MB 2025-02-14 21:59:31,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 21:59:31,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21349.37 MB 2025-02-14 21:59:31,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 21:59:31,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 21:59:31,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 21:59:31,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:31,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.27 MB 2025-02-14 21:59:31,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18269.96 MB 2025-02-14 21:59:31,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2292.70 MB 2025-02-14 21:59:31,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20554.19 MB 2025-02-14 21:59:31,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22913.48 MB 2025-02-14 21:59:31,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 21:59:31,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21349.37 MB 2025-02-14 21:59:31,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 21:59:31,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 21:59:31,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 21:59:31,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:31,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19121.08 MB 2025-02-14 21:59:31,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19547.02 MB 2025-02-14 21:59:31,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.95 MB 2025-02-14 21:59:31,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22913.48 MB 2025-02-14 21:59:31,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23142.07 MB 2025-02-14 21:59:31,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 228.59 MB 2025-02-14 21:59:31,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19939.92 MB 2025-02-14 21:59:31,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 21:59:31,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 21:59:31,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 21:59:31,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:31,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19776.18 MB 2025-02-14 21:59:31,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19996.02 MB 2025-02-14 21:59:31,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.84 MB 2025-02-14 21:59:31,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23142.07 MB 2025-02-14 21:59:31,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23142.07 MB 2025-02-14 21:59:31,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:59:31,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20073.63 MB 2025-02-14 21:59:31,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 21:59:31,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 21:59:31,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.76 seconds 2025-02-14 21:59:31,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:31,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13742.17 MB 2025-02-14 21:59:31,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20196.94 MB 2025-02-14 21:59:31,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6454.77 MB 2025-02-14 21:59:31,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47246.74 MB 2025-02-14 21:59:31,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23142.07 MB 2025-02-14 21:59:31,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24104.67 MB 2025-02-14 21:59:31,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20196.94 MB 2025-02-14 21:59:31,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 21:59:31,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 21:59:31,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 21:59:31,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:31,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14892.68 MB 2025-02-14 21:59:31,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17904.50 MB 2025-02-14 21:59:31,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.82 MB 2025-02-14 21:59:31,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23142.07 MB 2025-02-14 21:59:31,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23142.07 MB 2025-02-14 21:59:31,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 21:59:31,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18205.65 MB 2025-02-14 21:59:31,503 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 21:59:31,504 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 21:59:31,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 21:59:31,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 21:59:31,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 21:59:31,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 21:59:31,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.50 MB 2025-02-14 21:59:31,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26337.80 MB 2025-02-14 21:59:31,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 21:59:31,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23142.07 MB 2025-02-14 21:59:31,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31526.49 MB 2025-02-14 21:59:31,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 21:59:31,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26337.80 MB 2025-02-14 21:59:31,665 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 21:59:31,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:59:31,666 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 21:59:31,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:59:31,667 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 21:59:31,672 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 21:59:31,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 21:59:31,673 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 21:59:31,673 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:00:16,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:00:16,555 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:00:16,560 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:00:16,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:00:16,564 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:00:16,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:00:16,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:00:35,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:00:35,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:00:35,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.40 seconds 2025-02-14 22:00:35,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:35,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-14 22:00:35,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-14 22:00:35,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-14 22:00:35,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39910.90 MB 2025-02-14 22:00:35,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35169.24 MB 2025-02-14 22:00:35,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4741.66 MB 2025-02-14 22:00:35,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.17 MB 2025-02-14 22:00:36,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:00:36,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:00:36,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:00:36,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:36,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-14 22:00:36,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-14 22:00:36,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-14 22:00:36,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35169.24 MB 2025-02-14 22:00:36,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43945.82 MB 2025-02-14 22:00:36,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8776.58 MB 2025-02-14 22:00:36,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39374.65 MB 2025-02-14 22:00:37,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:00:37,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:00:37,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:00:37,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:37,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-14 22:00:37,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-14 22:00:37,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:00:37,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43945.82 MB 2025-02-14 22:00:37,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26499.61 MB 2025-02-14 22:00:37,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17446.21 MB 2025-02-14 22:00:37,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26864.62 MB 2025-02-14 22:00:37,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:00:37,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:00:37,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:00:37,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:37,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 22:00:37,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-14 22:00:37,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:00:37,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26499.61 MB 2025-02-14 22:00:37,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27443.33 MB 2025-02-14 22:00:37,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:00:37,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-14 22:00:38,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:00:38,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:00:38,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:00:38,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:38,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-14 22:00:38,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 22:00:38,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:00:38,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27443.33 MB 2025-02-14 22:00:38,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34049.36 MB 2025-02-14 22:00:38,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:00:38,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 22:00:38,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:00:38,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:00:38,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:00:38,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:38,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 22:00:38,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 22:00:38,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:00:38,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26499.61 MB 2025-02-14 22:00:38,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34049.36 MB 2025-02-14 22:00:38,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 22:00:38,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 22:00:38,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:00:38,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:00:38,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 22:00:38,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:38,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-14 22:00:38,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-14 22:00:38,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:00:38,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34049.36 MB 2025-02-14 22:00:38,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 22:00:38,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:00:38,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-14 22:00:38,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:00:38,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:00:38,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:00:38,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:38,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-14 22:00:38,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.77 MB 2025-02-14 22:00:38,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 22:00:38,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-14 22:00:38,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 22:00:38,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:00:38,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30172.74 MB 2025-02-14 22:00:38,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:00:38,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:00:38,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.83 seconds 2025-02-14 22:00:38,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:38,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-14 22:00:38,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30159.59 MB 2025-02-14 22:00:38,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12783.53 MB 2025-02-14 22:00:38,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39910.90 MB 2025-02-14 22:00:38,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 22:00:38,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5446.30 MB 2025-02-14 22:00:38,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30172.74 MB 2025-02-14 22:00:38,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:00:38,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:00:38,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:00:38,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:38,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30159.59 MB 2025-02-14 22:00:38,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22376.65 MB 2025-02-14 22:00:38,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7782.95 MB 2025-02-14 22:00:38,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-14 22:00:38,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 22:00:38,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:00:38,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32668.19 MB 2025-02-14 22:00:38,688 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 22:00:38,688 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 22:00:38,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:00:38,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:00:38,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:00:38,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:00:38,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22376.65 MB 2025-02-14 22:00:38,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30805.77 MB 2025-02-14 22:00:38,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 22:00:38,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-14 22:00:38,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38654.71 MB 2025-02-14 22:00:38,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 22:00:38,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30805.77 MB 2025-02-14 22:00:38,860 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 22:00:38,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:00:38,862 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:00:38,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:00:38,863 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:00:38,867 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:00:38,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:00:38,869 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:00:38,869 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 22:01:24,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:24,254 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:01:24,259 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:01:24,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:24,263 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1018, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:01:24,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:24,263 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1018, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:01:40,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:01:40,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:01:40,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.74 seconds 2025-02-14 22:01:40,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:40,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20062.29 MB 2025-02-14 22:01:40,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23665.20 MB 2025-02-14 22:01:40,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3602.91 MB 2025-02-14 22:01:40,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47034.93 MB 2025-02-14 22:01:40,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25910.31 MB 2025-02-14 22:01:40,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21124.61 MB 2025-02-14 22:01:40,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32478.87 MB 2025-02-14 22:01:40,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:01:40,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:01:40,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:01:40,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:40,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23665.20 MB 2025-02-14 22:01:40,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21071.16 MB 2025-02-14 22:01:40,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2594.04 MB 2025-02-14 22:01:40,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25910.31 MB 2025-02-14 22:01:40,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42658.17 MB 2025-02-14 22:01:40,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16747.86 MB 2025-02-14 22:01:40,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34795.78 MB 2025-02-14 22:01:42,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:01:42,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:01:42,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:01:42,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21071.16 MB 2025-02-14 22:01:42,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21602.00 MB 2025-02-14 22:01:42,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:01:42,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42658.17 MB 2025-02-14 22:01:42,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25140.66 MB 2025-02-14 22:01:42,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17517.51 MB 2025-02-14 22:01:42,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25581.59 MB 2025-02-14 22:01:42,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:01:42,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:01:42,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:01:42,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21602.00 MB 2025-02-14 22:01:42,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23491.54 MB 2025-02-14 22:01:42,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:01:42,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25140.66 MB 2025-02-14 22:01:42,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27028.09 MB 2025-02-14 22:01:42,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:01:42,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24908.97 MB 2025-02-14 22:01:42,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:01:42,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:01:42,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:01:42,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23491.54 MB 2025-02-14 22:01:42,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25733.39 MB 2025-02-14 22:01:42,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:01:42,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27028.09 MB 2025-02-14 22:01:42,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33162.26 MB 2025-02-14 22:01:42,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:01:42,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31277.68 MB 2025-02-14 22:01:42,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:01:42,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:01:42,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:01:42,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21602.00 MB 2025-02-14 22:01:42,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25733.39 MB 2025-02-14 22:01:42,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:01:42,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25140.66 MB 2025-02-14 22:01:42,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33162.26 MB 2025-02-14 22:01:42,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 22:01:42,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31277.68 MB 2025-02-14 22:01:42,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:01:42,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:01:42,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 22:01:42,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27266.94 MB 2025-02-14 22:01:42,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28033.94 MB 2025-02-14 22:01:42,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:01:42,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33162.26 MB 2025-02-14 22:01:42,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33577.50 MB 2025-02-14 22:01:42,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:01:42,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28741.73 MB 2025-02-14 22:01:42,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:01:42,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:01:42,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:01:42,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28446.83 MB 2025-02-14 22:01:42,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28676.85 MB 2025-02-14 22:01:42,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.02 MB 2025-02-14 22:01:42,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33577.50 MB 2025-02-14 22:01:42,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33577.50 MB 2025-02-14 22:01:42,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:01:42,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28885.71 MB 2025-02-14 22:01:42,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:01:42,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:01:42,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.16 seconds 2025-02-14 22:01:42,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16515.50 MB 2025-02-14 22:01:42,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28877.92 MB 2025-02-14 22:01:42,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12362.42 MB 2025-02-14 22:01:42,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47034.93 MB 2025-02-14 22:01:42,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33577.50 MB 2025-02-14 22:01:42,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13457.42 MB 2025-02-14 22:01:42,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28885.71 MB 2025-02-14 22:01:42,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:01:42,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:01:42,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:01:42,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28877.92 MB 2025-02-14 22:01:42,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21519.89 MB 2025-02-14 22:01:42,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7358.03 MB 2025-02-14 22:01:42,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33577.50 MB 2025-02-14 22:01:42,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33577.50 MB 2025-02-14 22:01:42,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:01:42,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31389.59 MB 2025-02-14 22:01:42,714 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:01:42,714 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:01:42,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:01:42,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:01:42,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:01:42,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:01:42,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21519.89 MB 2025-02-14 22:01:42,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.91 MB 2025-02-14 22:01:42,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:01:42,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33577.50 MB 2025-02-14 22:01:42,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44067.45 MB 2025-02-14 22:01:42,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 22:01:42,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29958.91 MB 2025-02-14 22:01:42,884 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:01:42,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:42,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:01:42,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:42,887 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:01:42,891 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:01:42,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:42,893 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:01:42,893 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:01:53,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:53,423 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:01:53,428 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:01:53,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:53,431 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1104, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:01:53,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:01:53,432 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1104, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:02:10,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:02:10,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:02:10,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.11 seconds 2025-02-14 22:02:10,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:10,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20661.56 MB 2025-02-14 22:02:10,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24568.55 MB 2025-02-14 22:02:10,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3906.99 MB 2025-02-14 22:02:10,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56652.46 MB 2025-02-14 22:02:10,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28313.65 MB 2025-02-14 22:02:10,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28338.81 MB 2025-02-14 22:02:10,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33530.31 MB 2025-02-14 22:02:10,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:02:10,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:02:10,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 22:02:10,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:10,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24568.55 MB 2025-02-14 22:02:10,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20330.70 MB 2025-02-14 22:02:10,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4237.84 MB 2025-02-14 22:02:10,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28313.65 MB 2025-02-14 22:02:10,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29131.54 MB 2025-02-14 22:02:10,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 817.89 MB 2025-02-14 22:02:10,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27870.20 MB 2025-02-14 22:02:11,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:02:11,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:02:11,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.11 seconds 2025-02-14 22:02:11,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:11,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20330.70 MB 2025-02-14 22:02:11,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20637.27 MB 2025-02-14 22:02:11,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.56 MB 2025-02-14 22:02:11,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29131.54 MB 2025-02-14 22:02:11,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25224.54 MB 2025-02-14 22:02:11,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3906.99 MB 2025-02-14 22:02:11,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24586.33 MB 2025-02-14 22:02:11,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:02:11,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:02:11,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:02:11,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:11,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20637.27 MB 2025-02-14 22:02:11,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21728.21 MB 2025-02-14 22:02:11,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1090.94 MB 2025-02-14 22:02:11,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25224.54 MB 2025-02-14 22:02:11,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25224.54 MB 2025-02-14 22:02:11,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:02:11,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22546.78 MB 2025-02-14 22:02:11,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:02:11,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:02:11,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 22:02:11,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:11,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21728.21 MB 2025-02-14 22:02:11,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23022.91 MB 2025-02-14 22:02:11,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1294.70 MB 2025-02-14 22:02:11,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25224.54 MB 2025-02-14 22:02:11,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27678.21 MB 2025-02-14 22:02:11,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2453.67 MB 2025-02-14 22:02:11,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26225.88 MB 2025-02-14 22:02:11,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:02:11,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:02:11,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 22:02:11,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:11,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20637.27 MB 2025-02-14 22:02:11,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23022.91 MB 2025-02-14 22:02:11,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2385.64 MB 2025-02-14 22:02:11,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25224.54 MB 2025-02-14 22:02:11,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27678.21 MB 2025-02-14 22:02:11,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2453.67 MB 2025-02-14 22:02:11,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26225.88 MB 2025-02-14 22:02:11,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:02:11,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:02:11,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:02:11,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:11,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23908.53 MB 2025-02-14 22:02:11,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24351.60 MB 2025-02-14 22:02:11,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 443.07 MB 2025-02-14 22:02:11,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27678.21 MB 2025-02-14 22:02:11,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-14 22:02:11,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 239.08 MB 2025-02-14 22:02:11,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24760.35 MB 2025-02-14 22:02:11,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:02:11,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:02:11,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:02:11,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:11,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.05 MB 2025-02-14 22:02:11,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24812.16 MB 2025-02-14 22:02:11,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.11 MB 2025-02-14 22:02:11,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 22:02:11,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-14 22:02:11,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:02:11,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24883.00 MB 2025-02-14 22:02:11,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:02:11,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:02:11,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.52 seconds 2025-02-14 22:02:11,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:11,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16815.13 MB 2025-02-14 22:02:11,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25013.23 MB 2025-02-14 22:02:11,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8198.10 MB 2025-02-14 22:02:11,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56652.46 MB 2025-02-14 22:02:11,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-14 22:02:11,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28735.18 MB 2025-02-14 22:02:11,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25013.23 MB 2025-02-14 22:02:12,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:02:12,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:02:12,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:02:12,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:12,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18008.06 MB 2025-02-14 22:02:12,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21022.09 MB 2025-02-14 22:02:12,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 22:02:12,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 22:02:12,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27917.29 MB 2025-02-14 22:02:12,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:02:12,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21323.46 MB 2025-02-14 22:02:12,238 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:02:12,238 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:02:12,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:02:12,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:02:12,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:02:12,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:12,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21022.09 MB 2025-02-14 22:02:12,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29461.11 MB 2025-02-14 22:02:12,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:02:12,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27917.29 MB 2025-02-14 22:02:12,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36307.99 MB 2025-02-14 22:02:12,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:02:12,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29461.11 MB 2025-02-14 22:02:12,400 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:02:12,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:12,402 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:02:12,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:12,403 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:02:12,407 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:02:12,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:12,408 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:02:12,409 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:02:40,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:40,966 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:02:40,974 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:02:40,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:40,980 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:02:40,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:40,982 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:02:43,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:02:43,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:02:43,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.83 seconds 2025-02-14 22:02:43,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:43,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-14 22:02:43,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-14 22:02:43,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-14 22:02:43,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48893.00 MB 2025-02-14 22:02:43,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20566.77 MB 2025-02-14 22:02:43,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28326.23 MB 2025-02-14 22:02:43,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23687.38 MB 2025-02-14 22:02:43,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:02:43,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:02:43,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:02:43,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:43,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-14 22:02:43,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15142.35 MB 2025-02-14 22:02:43,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.87 MB 2025-02-14 22:02:43,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20566.77 MB 2025-02-14 22:02:43,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20566.77 MB 2025-02-14 22:02:43,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:02:43,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.31 MB 2025-02-14 22:02:44,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:02:44,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:02:44,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 22:02:44,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:44,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15142.35 MB 2025-02-14 22:02:44,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15377.25 MB 2025-02-14 22:02:44,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 22:02:44,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20566.77 MB 2025-02-14 22:02:44,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19203.62 MB 2025-02-14 22:02:44,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1363.15 MB 2025-02-14 22:02:44,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19313.04 MB 2025-02-14 22:02:44,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:02:44,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:02:44,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:02:44,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:44,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-14 22:02:44,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16213.10 MB 2025-02-14 22:02:44,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 22:02:44,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19203.62 MB 2025-02-14 22:02:44,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19203.62 MB 2025-02-14 22:02:44,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:02:44,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16840.31 MB 2025-02-14 22:02:44,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:02:44,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:02:44,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:02:44,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:44,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16213.10 MB 2025-02-14 22:02:44,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-14 22:02:44,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 22:02:44,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19203.62 MB 2025-02-14 22:02:44,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20881.34 MB 2025-02-14 22:02:44,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1677.72 MB 2025-02-14 22:02:44,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19658.46 MB 2025-02-14 22:02:44,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:02:44,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:02:44,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:02:44,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:44,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-14 22:02:44,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-14 22:02:44,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 22:02:44,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19203.62 MB 2025-02-14 22:02:44,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20881.34 MB 2025-02-14 22:02:44,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1677.72 MB 2025-02-14 22:02:44,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19658.46 MB 2025-02-14 22:02:44,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:02:44,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:02:44,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:02:44,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:44,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17883.75 MB 2025-02-14 22:02:44,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18223.14 MB 2025-02-14 22:02:44,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 22:02:44,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20881.34 MB 2025-02-14 22:02:44,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21063.79 MB 2025-02-14 22:02:44,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 22:02:44,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18541.11 MB 2025-02-14 22:02:44,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:02:44,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:02:44,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:02:44,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:44,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18405.85 MB 2025-02-14 22:02:44,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18633.30 MB 2025-02-14 22:02:44,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.45 MB 2025-02-14 22:02:44,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21063.79 MB 2025-02-14 22:02:44,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21063.79 MB 2025-02-14 22:02:44,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:02:44,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18664.40 MB 2025-02-14 22:02:44,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:02:44,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:02:44,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.88 seconds 2025-02-14 22:02:44,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:44,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-14 22:02:44,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.37 MB 2025-02-14 22:02:44,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5242.02 MB 2025-02-14 22:02:44,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48893.00 MB 2025-02-14 22:02:44,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21063.79 MB 2025-02-14 22:02:44,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27829.21 MB 2025-02-14 22:02:44,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18834.37 MB 2025-02-14 22:02:45,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:02:45,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:02:45,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:02:45,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:45,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18834.37 MB 2025-02-14 22:02:45,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17544.34 MB 2025-02-14 22:02:45,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1290.03 MB 2025-02-14 22:02:45,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21063.79 MB 2025-02-14 22:02:45,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21063.79 MB 2025-02-14 22:02:45,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:02:45,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19068.80 MB 2025-02-14 22:02:45,155 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:02:45,155 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:02:45,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:02:45,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:02:45,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:02:45,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:02:45,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17544.34 MB 2025-02-14 22:02:45,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25983.37 MB 2025-02-14 22:02:45,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:02:45,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21063.79 MB 2025-02-14 22:02:45,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29454.50 MB 2025-02-14 22:02:45,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:02:45,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25983.37 MB 2025-02-14 22:02:45,316 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:02:45,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:45,318 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:02:45,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:45,319 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:02:45,323 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:02:45,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:02:45,324 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:02:45,324 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:03:41,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:03:41,366 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:03:41,370 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:03:41,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:03:41,374 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 474, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:03:41,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:03:41,375 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 474, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:03:48,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:03:48,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:03:48,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.24 seconds 2025-02-14 22:03:48,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:48,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16271.61 MB 2025-02-14 22:03:48,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17949.07 MB 2025-02-14 22:03:48,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1677.46 MB 2025-02-14 22:03:48,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42039.51 MB 2025-02-14 22:03:48,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22307.41 MB 2025-02-14 22:03:48,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19732.10 MB 2025-02-14 22:03:48,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26875.45 MB 2025-02-14 22:03:48,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:03:48,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:03:48,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 22:03:48,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:48,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17949.07 MB 2025-02-14 22:03:48,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18242.03 MB 2025-02-14 22:03:48,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.96 MB 2025-02-14 22:03:48,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22307.41 MB 2025-02-14 22:03:48,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27594.33 MB 2025-02-14 22:03:48,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5286.92 MB 2025-02-14 22:03:48,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25489.39 MB 2025-02-14 22:03:50,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:03:50,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:03:50,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.89 seconds 2025-02-14 22:03:50,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:50,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18242.03 MB 2025-02-14 22:03:50,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18772.87 MB 2025-02-14 22:03:50,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:03:50,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27594.33 MB 2025-02-14 22:03:50,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23722.98 MB 2025-02-14 22:03:50,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3871.34 MB 2025-02-14 22:03:50,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22751.42 MB 2025-02-14 22:03:50,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:03:50,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:03:50,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:03:50,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:50,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18772.87 MB 2025-02-14 22:03:50,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20662.40 MB 2025-02-14 22:03:50,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:03:50,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 22:03:50,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24666.70 MB 2025-02-14 22:03:50,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:03:50,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22079.83 MB 2025-02-14 22:03:50,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:03:50,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:03:50,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:03:50,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:50,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20662.40 MB 2025-02-14 22:03:50,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22904.26 MB 2025-02-14 22:03:50,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:03:50,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24666.70 MB 2025-02-14 22:03:50,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30329.01 MB 2025-02-14 22:03:50,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:03:50,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28448.54 MB 2025-02-14 22:03:50,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:03:50,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:03:50,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:03:50,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:50,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18772.87 MB 2025-02-14 22:03:50,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22904.26 MB 2025-02-14 22:03:50,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:03:50,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23722.98 MB 2025-02-14 22:03:50,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30329.01 MB 2025-02-14 22:03:50,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:03:50,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28448.54 MB 2025-02-14 22:03:50,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:03:50,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:03:50,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:03:50,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:50,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24437.80 MB 2025-02-14 22:03:50,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25204.80 MB 2025-02-14 22:03:50,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:03:50,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30329.01 MB 2025-02-14 22:03:50,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30746.35 MB 2025-02-14 22:03:50,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:03:50,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25912.59 MB 2025-02-14 22:03:50,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:03:50,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:03:50,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:03:50,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:50,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25617.69 MB 2025-02-14 22:03:50,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25847.50 MB 2025-02-14 22:03:50,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.81 MB 2025-02-14 22:03:50,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30746.35 MB 2025-02-14 22:03:50,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30746.35 MB 2025-02-14 22:03:50,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:03:50,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26070.47 MB 2025-02-14 22:03:50,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:03:50,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:03:50,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.57 seconds 2025-02-14 22:03:50,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:50,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14620.16 MB 2025-02-14 22:03:50,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.57 MB 2025-02-14 22:03:50,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11428.41 MB 2025-02-14 22:03:50,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42039.51 MB 2025-02-14 22:03:50,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30746.35 MB 2025-02-14 22:03:50,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11293.16 MB 2025-02-14 22:03:50,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26070.47 MB 2025-02-14 22:03:51,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:03:51,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:03:51,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:03:51,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:51,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.57 MB 2025-02-14 22:03:51,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19624.55 MB 2025-02-14 22:03:51,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6424.03 MB 2025-02-14 22:03:51,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30746.35 MB 2025-02-14 22:03:51,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30746.35 MB 2025-02-14 22:03:51,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:03:51,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28560.24 MB 2025-02-14 22:03:51,232 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:03:51,232 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 22:03:51,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:03:51,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:03:51,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:03:51,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:03:51,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19624.55 MB 2025-02-14 22:03:51,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28063.57 MB 2025-02-14 22:03:51,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:03:51,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30746.35 MB 2025-02-14 22:03:51,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41236.30 MB 2025-02-14 22:03:51,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 22:03:51,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28063.57 MB 2025-02-14 22:03:51,395 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:03:51,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:03:51,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:03:51,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:03:51,398 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:03:51,402 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:03:51,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:03:51,403 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:03:51,404 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 22:04:03,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:03,678 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:04:03,683 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:04:03,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:03,687 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1385, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:04:03,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:03,687 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1385, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:04:25,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:04:25,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:04:25,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.39 seconds 2025-02-14 22:04:25,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:25,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22619.61 MB 2025-02-14 22:04:25,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27521.05 MB 2025-02-14 22:04:25,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4901.44 MB 2025-02-14 22:04:25,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53821.31 MB 2025-02-14 22:04:25,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35599.16 MB 2025-02-14 22:04:25,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18222.15 MB 2025-02-14 22:04:25,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36394.33 MB 2025-02-14 22:04:25,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:04:25,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:04:25,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:04:25,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:25,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27521.05 MB 2025-02-14 22:04:25,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22978.03 MB 2025-02-14 22:04:25,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4543.01 MB 2025-02-14 22:04:25,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35599.16 MB 2025-02-14 22:04:25,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47242.54 MB 2025-02-14 22:04:25,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11643.39 MB 2025-02-14 22:04:25,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42162.09 MB 2025-02-14 22:04:27,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:04:27,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:04:27,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:04:27,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22978.03 MB 2025-02-14 22:04:27,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.88 MB 2025-02-14 22:04:27,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:04:27,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47242.54 MB 2025-02-14 22:04:27,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30696.01 MB 2025-02-14 22:04:27,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16546.53 MB 2025-02-14 22:04:27,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27487.42 MB 2025-02-14 22:04:27,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:04:27,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:04:27,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:04:27,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.88 MB 2025-02-14 22:04:27,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.41 MB 2025-02-14 22:04:27,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:04:27,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 22:04:27,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30696.01 MB 2025-02-14 22:04:27,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:27,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26815.84 MB 2025-02-14 22:04:27,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:04:27,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:04:27,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:04:27,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.41 MB 2025-02-14 22:04:27,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27640.27 MB 2025-02-14 22:04:27,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:04:27,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 22:04:27,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35414.61 MB 2025-02-14 22:04:27,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 22:04:27,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33184.55 MB 2025-02-14 22:04:27,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:04:27,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:04:27,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:04:27,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.88 MB 2025-02-14 22:04:27,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27640.27 MB 2025-02-14 22:04:27,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:04:27,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30696.01 MB 2025-02-14 22:04:27,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35414.61 MB 2025-02-14 22:04:27,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 22:04:27,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33184.55 MB 2025-02-14 22:04:27,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:04:27,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:04:27,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:04:27,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29173.81 MB 2025-02-14 22:04:27,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29940.81 MB 2025-02-14 22:04:27,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:04:27,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35414.61 MB 2025-02-14 22:04:27,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 22:04:27,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:04:27,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30648.60 MB 2025-02-14 22:04:27,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:04:27,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:04:27,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:04:27,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30353.70 MB 2025-02-14 22:04:27,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30582.61 MB 2025-02-14 22:04:27,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 22:04:27,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35831.94 MB 2025-02-14 22:04:27,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 22:04:27,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:27,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30825.83 MB 2025-02-14 22:04:27,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:04:27,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:04:27,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.81 seconds 2025-02-14 22:04:27,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17794.16 MB 2025-02-14 22:04:27,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30783.44 MB 2025-02-14 22:04:27,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12989.28 MB 2025-02-14 22:04:27,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53821.31 MB 2025-02-14 22:04:27,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 22:04:27,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17989.37 MB 2025-02-14 22:04:27,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30825.83 MB 2025-02-14 22:04:27,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:04:27,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:04:27,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:04:27,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30783.44 MB 2025-02-14 22:04:27,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22794.74 MB 2025-02-14 22:04:27,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7988.70 MB 2025-02-14 22:04:27,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35831.94 MB 2025-02-14 22:04:27,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35831.94 MB 2025-02-14 22:04:27,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:27,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33292.03 MB 2025-02-14 22:04:27,781 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 22:04:27,781 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:04:27,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:04:27,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:04:27,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:04:27,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:27,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22794.74 MB 2025-02-14 22:04:27,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31223.86 MB 2025-02-14 22:04:27,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 22:04:27,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35831.94 MB 2025-02-14 22:04:27,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40022.05 MB 2025-02-14 22:04:27,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 22:04:27,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31223.86 MB 2025-02-14 22:04:27,943 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 22:04:27,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:27,944 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:04:27,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:27,945 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:04:27,949 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:04:27,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:27,951 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:04:27,951 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:04:37,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:37,873 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:04:37,880 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:04:37,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:37,887 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 235, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:04:37,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:37,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 235, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:04:41,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:04:41,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:04:41,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.73 seconds 2025-02-14 22:04:41,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:41,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.22 MB 2025-02-14 22:04:41,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15437.88 MB 2025-02-14 22:04:41,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.65 MB 2025-02-14 22:04:41,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48402.27 MB 2025-02-14 22:04:41,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23251.12 MB 2025-02-14 22:04:41,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25151.14 MB 2025-02-14 22:04:41,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24304.09 MB 2025-02-14 22:04:41,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:04:41,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:04:41,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:04:41,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:41,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15437.88 MB 2025-02-14 22:04:41,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15840.74 MB 2025-02-14 22:04:41,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.87 MB 2025-02-14 22:04:41,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23251.12 MB 2025-02-14 22:04:41,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23251.12 MB 2025-02-14 22:04:41,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:41,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18777.63 MB 2025-02-14 22:04:42,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:04:42,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:04:42,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.14 seconds 2025-02-14 22:04:42,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:42,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15840.74 MB 2025-02-14 22:04:42,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16152.61 MB 2025-02-14 22:04:42,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.87 MB 2025-02-14 22:04:42,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23251.12 MB 2025-02-14 22:04:42,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23251.12 MB 2025-02-14 22:04:42,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:42,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20095.33 MB 2025-02-14 22:04:42,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:04:42,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:04:42,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:04:42,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:42,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16152.61 MB 2025-02-14 22:04:42,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17262.45 MB 2025-02-14 22:04:42,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.83 MB 2025-02-14 22:04:42,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23251.12 MB 2025-02-14 22:04:42,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23251.12 MB 2025-02-14 22:04:42,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:42,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18095.19 MB 2025-02-14 22:04:42,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:04:42,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:04:42,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 22:04:42,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:42,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17262.45 MB 2025-02-14 22:04:42,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18579.56 MB 2025-02-14 22:04:42,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1317.12 MB 2025-02-14 22:04:42,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23251.12 MB 2025-02-14 22:04:42,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23251.12 MB 2025-02-14 22:04:42,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:42,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21836.80 MB 2025-02-14 22:04:42,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:04:42,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:04:42,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 22:04:42,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:42,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16152.61 MB 2025-02-14 22:04:42,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18579.56 MB 2025-02-14 22:04:42,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2426.95 MB 2025-02-14 22:04:42,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23251.12 MB 2025-02-14 22:04:42,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23251.12 MB 2025-02-14 22:04:42,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:42,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21836.80 MB 2025-02-14 22:04:43,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:04:43,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:04:43,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:04:43,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:43,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19480.52 MB 2025-02-14 22:04:43,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19931.13 MB 2025-02-14 22:04:43,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 450.61 MB 2025-02-14 22:04:43,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23251.12 MB 2025-02-14 22:04:43,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23496.49 MB 2025-02-14 22:04:43,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 245.37 MB 2025-02-14 22:04:43,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20346.96 MB 2025-02-14 22:04:43,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:04:43,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:04:43,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:04:43,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:43,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20173.71 MB 2025-02-14 22:04:43,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20382.21 MB 2025-02-14 22:04:43,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.50 MB 2025-02-14 22:04:43,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23496.49 MB 2025-02-14 22:04:43,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23496.49 MB 2025-02-14 22:04:43,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:04:43,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20447.97 MB 2025-02-14 22:04:43,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:04:43,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:04:43,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.15 seconds 2025-02-14 22:04:43,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:43,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13787.47 MB 2025-02-14 22:04:43,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20583.28 MB 2025-02-14 22:04:43,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6795.82 MB 2025-02-14 22:04:43,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48402.27 MB 2025-02-14 22:04:43,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23496.49 MB 2025-02-14 22:04:43,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24905.78 MB 2025-02-14 22:04:43,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20583.28 MB 2025-02-14 22:04:43,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:04:43,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:04:43,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:04:43,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:43,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20583.28 MB 2025-02-14 22:04:43,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23597.31 MB 2025-02-14 22:04:43,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 22:04:43,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23496.49 MB 2025-02-14 22:04:43,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25241.32 MB 2025-02-14 22:04:43,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1744.83 MB 2025-02-14 22:04:43,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23898.94 MB 2025-02-14 22:04:43,326 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:04:43,326 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:04:43,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:04:43,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:04:43,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:04:43,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:04:43,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18013.17 MB 2025-02-14 22:04:43,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26452.19 MB 2025-02-14 22:04:43,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:04:43,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25241.32 MB 2025-02-14 22:04:43,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33632.03 MB 2025-02-14 22:04:43,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:04:43,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26452.19 MB 2025-02-14 22:04:43,488 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:04:43,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:43,490 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:04:43,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:43,491 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:04:43,495 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:04:43,496 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:04:43,496 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:04:43,497 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:05:32,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:05:32,778 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:05:32,784 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:05:32,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:05:32,796 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 167, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:05:32,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:05:32,798 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 167, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:05:35,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:05:35,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:05:35,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.63 seconds 2025-02-14 22:05:35,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:35,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14132.39 MB 2025-02-14 22:05:35,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14723.39 MB 2025-02-14 22:05:35,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 591.00 MB 2025-02-14 22:05:35,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46217.04 MB 2025-02-14 22:05:35,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20696.79 MB 2025-02-14 22:05:35,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25520.24 MB 2025-02-14 22:05:35,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23604.78 MB 2025-02-14 22:05:35,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:05:35,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:05:35,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:05:35,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:35,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14723.39 MB 2025-02-14 22:05:35,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14967.59 MB 2025-02-14 22:05:35,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 244.20 MB 2025-02-14 22:05:35,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20696.79 MB 2025-02-14 22:05:35,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20696.79 MB 2025-02-14 22:05:35,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:05:35,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17009.64 MB 2025-02-14 22:05:36,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:05:36,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:05:36,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 22:05:36,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14967.59 MB 2025-02-14 22:05:36,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15181.26 MB 2025-02-14 22:05:36,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 22:05:36,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20696.79 MB 2025-02-14 22:05:36,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19058.92 MB 2025-02-14 22:05:36,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1637.88 MB 2025-02-14 22:05:36,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19138.28 MB 2025-02-14 22:05:36,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:05:36,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:05:36,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:05:36,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15181.19 MB 2025-02-14 22:05:36,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15941.54 MB 2025-02-14 22:05:36,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 22:05:36,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-14 22:05:36,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19058.92 MB 2025-02-14 22:05:36,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:05:36,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16512.07 MB 2025-02-14 22:05:36,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:05:36,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:05:36,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:05:36,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15941.54 MB 2025-02-14 22:05:36,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16843.93 MB 2025-02-14 22:05:36,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 22:05:36,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-14 22:05:36,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20203.96 MB 2025-02-14 22:05:36,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 22:05:36,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19075.47 MB 2025-02-14 22:05:36,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:05:36,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:05:36,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 22:05:36,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15181.19 MB 2025-02-14 22:05:36,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16843.93 MB 2025-02-14 22:05:36,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 22:05:36,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19058.92 MB 2025-02-14 22:05:36,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20203.96 MB 2025-02-14 22:05:36,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 22:05:36,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19075.47 MB 2025-02-14 22:05:36,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:05:36,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:05:36,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:05:36,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17461.18 MB 2025-02-14 22:05:36,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17769.90 MB 2025-02-14 22:05:36,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 22:05:36,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20203.96 MB 2025-02-14 22:05:36,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20367.54 MB 2025-02-14 22:05:36,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 22:05:36,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18062.60 MB 2025-02-14 22:05:36,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:05:36,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:05:36,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:05:36,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17936.10 MB 2025-02-14 22:05:36,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18165.24 MB 2025-02-14 22:05:36,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.14 MB 2025-02-14 22:05:36,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20367.54 MB 2025-02-14 22:05:36,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20367.54 MB 2025-02-14 22:05:36,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:05:36,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18181.67 MB 2025-02-14 22:05:36,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:05:36,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:05:36,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.71 seconds 2025-02-14 22:05:36,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13550.55 MB 2025-02-14 22:05:36,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18366.31 MB 2025-02-14 22:05:36,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4815.76 MB 2025-02-14 22:05:36,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46217.04 MB 2025-02-14 22:05:36,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20367.54 MB 2025-02-14 22:05:36,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25849.50 MB 2025-02-14 22:05:36,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18366.31 MB 2025-02-14 22:05:36,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:05:36,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:05:36,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 22:05:36,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18366.31 MB 2025-02-14 22:05:36,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17427.03 MB 2025-02-14 22:05:36,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -939.28 MB 2025-02-14 22:05:36,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20367.54 MB 2025-02-14 22:05:36,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20367.54 MB 2025-02-14 22:05:36,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:05:36,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19170.04 MB 2025-02-14 22:05:36,821 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:05:36,822 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 22:05:36,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:05:36,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:05:36,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:05:36,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:05:36,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17427.03 MB 2025-02-14 22:05:36,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25866.05 MB 2025-02-14 22:05:36,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:05:36,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20367.54 MB 2025-02-14 22:05:36,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30857.49 MB 2025-02-14 22:05:36,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 22:05:36,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25866.05 MB 2025-02-14 22:05:37,080 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:05:37,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:05:37,082 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:05:37,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:05:37,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:05:37,091 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:05:37,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:05:37,094 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:05:37,094 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 22:06:25,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:06:25,707 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:06:25,712 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:06:25,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:06:25,716 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:06:25,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:06:25,717 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:06:44,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:06:44,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:06:44,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.66 seconds 2025-02-14 22:06:44,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:44,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-14 22:06:44,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-14 22:06:44,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 22:06:44,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43442.50 MB 2025-02-14 22:06:44,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32908.51 MB 2025-02-14 22:06:44,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10533.99 MB 2025-02-14 22:06:44,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-14 22:06:44,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:06:44,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:06:44,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:06:44,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:44,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-14 22:06:44,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22104.65 MB 2025-02-14 22:06:44,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-14 22:06:44,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32908.51 MB 2025-02-14 22:06:44,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44604.33 MB 2025-02-14 22:06:44,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11695.82 MB 2025-02-14 22:06:44,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38616.30 MB 2025-02-14 22:06:46,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:06:46,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:06:46,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:06:46,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:46,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22104.65 MB 2025-02-14 22:06:46,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22635.50 MB 2025-02-14 22:06:46,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:06:46,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44604.33 MB 2025-02-14 22:06:46,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30016.54 MB 2025-02-14 22:06:46,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14587.79 MB 2025-02-14 22:06:46,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26614.04 MB 2025-02-14 22:06:46,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:06:46,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:06:46,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:06:46,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:46,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-14 22:06:46,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.03 MB 2025-02-14 22:06:46,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:06:46,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30016.54 MB 2025-02-14 22:06:46,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30016.54 MB 2025-02-14 22:06:46,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:06:46,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.46 MB 2025-02-14 22:06:46,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:06:46,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:06:46,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:06:46,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:46,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24525.03 MB 2025-02-14 22:06:46,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-14 22:06:46,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:06:46,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30016.54 MB 2025-02-14 22:06:46,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34263.27 MB 2025-02-14 22:06:46,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 22:06:46,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-14 22:06:46,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:06:46,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:06:46,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:06:46,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:46,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-14 22:06:46,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-14 22:06:46,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:06:46,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30016.54 MB 2025-02-14 22:06:46,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34263.27 MB 2025-02-14 22:06:46,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 22:06:46,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-14 22:06:46,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:06:46,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:06:46,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 22:06:46,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:46,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28300.43 MB 2025-02-14 22:06:46,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29067.43 MB 2025-02-14 22:06:46,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:06:46,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34263.27 MB 2025-02-14 22:06:46,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34680.60 MB 2025-02-14 22:06:46,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:06:46,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29775.22 MB 2025-02-14 22:06:46,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:06:46,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:06:46,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:06:46,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:46,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29480.32 MB 2025-02-14 22:06:46,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29708.84 MB 2025-02-14 22:06:46,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-14 22:06:46,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34680.60 MB 2025-02-14 22:06:46,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34680.60 MB 2025-02-14 22:06:46,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:06:46,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.88 MB 2025-02-14 22:06:46,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:06:46,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:06:46,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.08 seconds 2025-02-14 22:06:46,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:46,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-14 22:06:46,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29909.32 MB 2025-02-14 22:06:46,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12700.49 MB 2025-02-14 22:06:46,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43442.50 MB 2025-02-14 22:06:46,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34680.60 MB 2025-02-14 22:06:46,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8761.90 MB 2025-02-14 22:06:46,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.88 MB 2025-02-14 22:06:47,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:06:47,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:06:47,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:06:47,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:47,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29909.32 MB 2025-02-14 22:06:47,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22203.37 MB 2025-02-14 22:06:47,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7705.95 MB 2025-02-14 22:06:47,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34680.60 MB 2025-02-14 22:06:47,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34680.60 MB 2025-02-14 22:06:47,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:06:47,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32413.00 MB 2025-02-14 22:06:47,079 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 22:06:47,079 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:06:47,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:06:47,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:06:47,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:06:47,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:06:47,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22203.37 MB 2025-02-14 22:06:47,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30615.79 MB 2025-02-14 22:06:47,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 22:06:47,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34680.60 MB 2025-02-14 22:06:47,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43044.04 MB 2025-02-14 22:06:47,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 22:06:47,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30615.79 MB 2025-02-14 22:06:47,251 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 22:06:47,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:06:47,253 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:06:47,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:06:47,254 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:06:47,259 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:06:47,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:06:47,260 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:06:47,260 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:07:00,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:00,111 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:07:00,116 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:07:00,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:00,119 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1244, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:07:00,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:00,120 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1244, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:07:19,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:07:19,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:07:19,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.16 seconds 2025-02-14 22:07:19,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:19,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21637.10 MB 2025-02-14 22:07:19,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26039.54 MB 2025-02-14 22:07:19,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4402.45 MB 2025-02-14 22:07:19,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51407.49 MB 2025-02-14 22:07:19,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35062.28 MB 2025-02-14 22:07:19,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16345.20 MB 2025-02-14 22:07:19,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34958.84 MB 2025-02-14 22:07:19,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:07:19,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:07:19,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:07:19,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:19,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26039.54 MB 2025-02-14 22:07:19,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22245.02 MB 2025-02-14 22:07:19,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3794.53 MB 2025-02-14 22:07:19,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35062.28 MB 2025-02-14 22:07:19,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43702.55 MB 2025-02-14 22:07:19,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8640.27 MB 2025-02-14 22:07:19,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39075.22 MB 2025-02-14 22:07:21,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:07:21,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:07:21,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:07:21,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22245.02 MB 2025-02-14 22:07:21,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22775.86 MB 2025-02-14 22:07:21,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:07:21,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43702.55 MB 2025-02-14 22:07:21,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26476.54 MB 2025-02-14 22:07:21,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17226.01 MB 2025-02-14 22:07:21,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26755.45 MB 2025-02-14 22:07:21,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:07:21,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:07:21,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:21,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22775.86 MB 2025-02-14 22:07:21,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24665.39 MB 2025-02-14 22:07:21,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:07:21,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26476.54 MB 2025-02-14 22:07:21,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27420.26 MB 2025-02-14 22:07:21,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:07:21,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26082.82 MB 2025-02-14 22:07:21,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:07:21,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:07:21,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:07:21,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24665.39 MB 2025-02-14 22:07:21,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26907.25 MB 2025-02-14 22:07:21,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:07:21,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27420.26 MB 2025-02-14 22:07:21,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34026.29 MB 2025-02-14 22:07:21,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:07:21,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32451.53 MB 2025-02-14 22:07:21,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:07:21,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:07:21,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:07:21,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22775.86 MB 2025-02-14 22:07:21,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26907.25 MB 2025-02-14 22:07:21,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:07:21,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26476.54 MB 2025-02-14 22:07:21,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34026.29 MB 2025-02-14 22:07:21,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 22:07:21,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32451.53 MB 2025-02-14 22:07:21,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:07:21,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:07:21,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:07:21,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28440.79 MB 2025-02-14 22:07:21,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29207.79 MB 2025-02-14 22:07:21,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:07:21,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34026.29 MB 2025-02-14 22:07:21,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:07:21,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:07:21,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29915.58 MB 2025-02-14 22:07:21,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:07:21,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:07:21,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:07:21,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29620.68 MB 2025-02-14 22:07:21,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29849.55 MB 2025-02-14 22:07:21,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-14 22:07:21,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 22:07:21,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:07:21,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:21,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30089.91 MB 2025-02-14 22:07:21,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:07:21,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:07:21,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.57 seconds 2025-02-14 22:07:21,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17302.90 MB 2025-02-14 22:07:21,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30050.32 MB 2025-02-14 22:07:21,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12747.42 MB 2025-02-14 22:07:21,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51407.49 MB 2025-02-14 22:07:21,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:07:21,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16963.86 MB 2025-02-14 22:07:21,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30089.91 MB 2025-02-14 22:07:21,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:07:21,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:07:21,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:07:21,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30050.32 MB 2025-02-14 22:07:21,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22302.72 MB 2025-02-14 22:07:21,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7747.60 MB 2025-02-14 22:07:21,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 22:07:21,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:07:21,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:21,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32558.30 MB 2025-02-14 22:07:21,976 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 22:07:21,977 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:07:21,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:07:21,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:07:21,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:07:21,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:21,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22302.72 MB 2025-02-14 22:07:21,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30729.22 MB 2025-02-14 22:07:21,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 22:07:21,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 22:07:21,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42821.75 MB 2025-02-14 22:07:21,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 22:07:21,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30729.22 MB 2025-02-14 22:07:22,142 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 22:07:22,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:22,144 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:07:22,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:22,145 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:07:22,149 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:07:22,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:22,151 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:07:22,151 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:07:32,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:32,803 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:07:32,808 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:07:32,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:32,811 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:07:32,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:32,812 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:07:37,217 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:07:37,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:07:37,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.40 seconds 2025-02-14 22:07:37,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.70 MB 2025-02-14 22:07:37,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15942.22 MB 2025-02-14 22:07:37,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1001.52 MB 2025-02-14 22:07:37,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55387.88 MB 2025-02-14 22:07:37,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21887.98 MB 2025-02-14 22:07:37,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33499.91 MB 2025-02-14 22:07:37,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24865.05 MB 2025-02-14 22:07:37,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:07:37,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:07:37,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:37,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15942.22 MB 2025-02-14 22:07:37,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14580.40 MB 2025-02-14 22:07:37,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1361.82 MB 2025-02-14 22:07:37,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21887.98 MB 2025-02-14 22:07:37,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21887.98 MB 2025-02-14 22:07:37,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:37,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16262.13 MB 2025-02-14 22:07:37,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:07:37,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:07:37,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:07:37,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14580.40 MB 2025-02-14 22:07:37,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14606.94 MB 2025-02-14 22:07:37,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 26.54 MB 2025-02-14 22:07:37,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21887.98 MB 2025-02-14 22:07:37,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 22:07:37,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 22:07:37,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15857.66 MB 2025-02-14 22:07:37,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:07:37,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:07:37,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:07:37,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.87 MB 2025-02-14 22:07:37,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14701.33 MB 2025-02-14 22:07:37,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 94.45 MB 2025-02-14 22:07:37,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 22:07:37,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 22:07:37,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:37,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14772.21 MB 2025-02-14 22:07:37,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:07:37,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:07:37,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:37,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14701.33 MB 2025-02-14 22:07:37,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14814.07 MB 2025-02-14 22:07:37,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 112.74 MB 2025-02-14 22:07:37,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 22:07:37,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 22:07:37,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:37,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15092.40 MB 2025-02-14 22:07:37,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:07:37,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:07:37,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:07:37,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.87 MB 2025-02-14 22:07:37,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14814.07 MB 2025-02-14 22:07:37,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.20 MB 2025-02-14 22:07:37,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 22:07:37,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 22:07:37,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:37,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15092.40 MB 2025-02-14 22:07:37,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:07:37,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:07:37,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:37,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14891.93 MB 2025-02-14 22:07:37,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14930.28 MB 2025-02-14 22:07:37,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 38.35 MB 2025-02-14 22:07:37,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 22:07:37,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20963.13 MB 2025-02-14 22:07:37,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18.87 MB 2025-02-14 22:07:37,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14981.07 MB 2025-02-14 22:07:37,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:07:37,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:07:37,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:07:37,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14950.93 MB 2025-02-14 22:07:37,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14976.55 MB 2025-02-14 22:07:37,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.62 MB 2025-02-14 22:07:37,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20963.13 MB 2025-02-14 22:07:37,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20963.13 MB 2025-02-14 22:07:37,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:37,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14976.55 MB 2025-02-14 22:07:37,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:07:37,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:07:37,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.56 seconds 2025-02-14 22:07:37,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13954.70 MB 2025-02-14 22:07:37,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15023.69 MB 2025-02-14 22:07:37,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1068.99 MB 2025-02-14 22:07:37,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55387.88 MB 2025-02-14 22:07:37,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20963.13 MB 2025-02-14 22:07:37,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34424.75 MB 2025-02-14 22:07:37,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15023.69 MB 2025-02-14 22:07:37,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:07:37,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:07:37,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:07:37,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15023.69 MB 2025-02-14 22:07:37,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15730.41 MB 2025-02-14 22:07:37,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 706.72 MB 2025-02-14 22:07:37,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20963.13 MB 2025-02-14 22:07:37,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20967.33 MB 2025-02-14 22:07:37,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 22:07:37,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15801.07 MB 2025-02-14 22:07:37,448 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1903, cut from 1905 2025-02-14 22:07:37,448 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 22:07:37,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:07:37,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:07:37,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:37,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:37,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14782.59 MB 2025-02-14 22:07:37,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16760.79 MB 2025-02-14 22:07:37,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1978.20 MB 2025-02-14 22:07:37,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20967.33 MB 2025-02-14 22:07:37,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20967.33 MB 2025-02-14 22:07:37,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:37,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16760.79 MB 2025-02-14 22:07:37,487 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1695] 2025-02-14 22:07:37,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:37,488 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:07:37,489 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:37,489 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:07:37,494 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:07:37,495 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:37,495 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:07:37,495 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 22:07:44,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:44,135 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:07:44,143 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:07:44,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:44,149 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 194, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:07:44,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:44,151 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 194, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:07:47,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:07:47,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:07:47,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.08 seconds 2025-02-14 22:07:47,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:47,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14320.53 MB 2025-02-14 22:07:47,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15007.08 MB 2025-02-14 22:07:47,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 686.56 MB 2025-02-14 22:07:47,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20967.33 MB 2025-02-14 22:07:47,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 22:07:47,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23.07 MB 2025-02-14 22:07:47,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24018.39 MB 2025-02-14 22:07:47,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:07:47,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:07:47,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:47,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:47,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15007.08 MB 2025-02-14 22:07:47,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15332.70 MB 2025-02-14 22:07:47,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 325.61 MB 2025-02-14 22:07:47,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 22:07:47,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20944.26 MB 2025-02-14 22:07:47,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:47,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17725.11 MB 2025-02-14 22:07:48,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:07:48,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:07:48,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-14 22:07:48,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15332.70 MB 2025-02-14 22:07:48,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15588.83 MB 2025-02-14 22:07:48,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 22:07:48,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20944.26 MB 2025-02-14 22:07:48,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20000.54 MB 2025-02-14 22:07:48,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 22:07:48,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19587.28 MB 2025-02-14 22:07:48,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:07:48,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:07:48,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:48,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15588.76 MB 2025-02-14 22:07:48,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16500.24 MB 2025-02-14 22:07:48,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 22:07:48,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20000.54 MB 2025-02-14 22:07:48,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20000.54 MB 2025-02-14 22:07:48,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:48,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17184.15 MB 2025-02-14 22:07:48,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:07:48,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:07:48,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:07:48,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16500.24 MB 2025-02-14 22:07:48,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17581.97 MB 2025-02-14 22:07:48,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-14 22:07:48,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20000.54 MB 2025-02-14 22:07:48,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21600.67 MB 2025-02-14 22:07:48,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1600.13 MB 2025-02-14 22:07:48,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20257.97 MB 2025-02-14 22:07:48,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:07:48,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:07:48,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:07:48,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15588.76 MB 2025-02-14 22:07:48,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17581.97 MB 2025-02-14 22:07:48,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-14 22:07:48,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20000.54 MB 2025-02-14 22:07:48,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21600.67 MB 2025-02-14 22:07:48,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1600.13 MB 2025-02-14 22:07:48,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20257.97 MB 2025-02-14 22:07:48,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:07:48,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:07:48,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:07:48,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18321.90 MB 2025-02-14 22:07:48,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18692.90 MB 2025-02-14 22:07:48,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.00 MB 2025-02-14 22:07:48,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21600.67 MB 2025-02-14 22:07:48,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21799.90 MB 2025-02-14 22:07:48,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 22:07:48,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19037.79 MB 2025-02-14 22:07:48,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:07:48,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:07:48,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:07:48,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18892.13 MB 2025-02-14 22:07:48,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19120.14 MB 2025-02-14 22:07:48,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.02 MB 2025-02-14 22:07:48,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21799.90 MB 2025-02-14 22:07:48,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21799.90 MB 2025-02-14 22:07:48,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:48,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19157.66 MB 2025-02-14 22:07:48,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:07:48,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:07:48,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.24 seconds 2025-02-14 22:07:48,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13644.62 MB 2025-02-14 22:07:48,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19321.21 MB 2025-02-14 22:07:48,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5676.60 MB 2025-02-14 22:07:48,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20967.33 MB 2025-02-14 22:07:48,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21801.99 MB 2025-02-14 22:07:48,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 834.67 MB 2025-02-14 22:07:48,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19321.21 MB 2025-02-14 22:07:48,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:07:48,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:07:48,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:07:48,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19321.21 MB 2025-02-14 22:07:48,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17673.03 MB 2025-02-14 22:07:48,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1648.18 MB 2025-02-14 22:07:48,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21801.99 MB 2025-02-14 22:07:48,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21801.99 MB 2025-02-14 22:07:48,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:07:48,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19321.21 MB 2025-02-14 22:07:48,677 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:07:48,677 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 22:07:48,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:07:48,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:07:48,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:07:48,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:07:48,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17673.03 MB 2025-02-14 22:07:48,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26112.05 MB 2025-02-14 22:07:48,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:07:48,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21801.99 MB 2025-02-14 22:07:48,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30192.70 MB 2025-02-14 22:07:48,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:07:48,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26112.05 MB 2025-02-14 22:07:48,841 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:07:48,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:48,843 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:07:48,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:48,844 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:07:48,848 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:07:48,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:07:48,849 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:07:48,849 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 22:08:59,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:08:59,620 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:08:59,625 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:08:59,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:08:59,628 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 109, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:08:59,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:08:59,629 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 109, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:09:01,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:09:01,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:09:01,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.71 seconds 2025-02-14 22:09:01,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:01,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13728.24 MB 2025-02-14 22:09:01,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14113.98 MB 2025-02-14 22:09:01,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 385.74 MB 2025-02-14 22:09:01,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42777.71 MB 2025-02-14 22:09:01,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21099.45 MB 2025-02-14 22:09:01,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21678.26 MB 2025-02-14 22:09:01,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22973.11 MB 2025-02-14 22:09:01,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:09:01,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:09:01,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:09:01,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:01,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14113.98 MB 2025-02-14 22:09:01,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14300.87 MB 2025-02-14 22:09:01,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 186.89 MB 2025-02-14 22:09:01,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21099.45 MB 2025-02-14 22:09:01,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21099.45 MB 2025-02-14 22:09:01,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:09:01,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14879.55 MB 2025-02-14 22:09:01,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:09:01,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:09:01,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.53 seconds 2025-02-14 22:09:01,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:01,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14300.87 MB 2025-02-14 22:09:01,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14445.53 MB 2025-02-14 22:09:01,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 144.65 MB 2025-02-14 22:09:01,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21099.45 MB 2025-02-14 22:09:01,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21099.45 MB 2025-02-14 22:09:01,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:09:01,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18385.59 MB 2025-02-14 22:09:01,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:09:01,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:09:01,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:09:01,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:01,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.46 MB 2025-02-14 22:09:01,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14960.23 MB 2025-02-14 22:09:01,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 514.77 MB 2025-02-14 22:09:01,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21099.45 MB 2025-02-14 22:09:01,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21099.45 MB 2025-02-14 22:09:01,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:09:01,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15346.49 MB 2025-02-14 22:09:02,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:09:02,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:09:02,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 22:09:02,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:02,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14960.23 MB 2025-02-14 22:09:02,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15585.47 MB 2025-02-14 22:09:02,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 625.24 MB 2025-02-14 22:09:02,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21099.45 MB 2025-02-14 22:09:02,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21099.45 MB 2025-02-14 22:09:02,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:09:02,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17081.96 MB 2025-02-14 22:09:02,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:09:02,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:09:02,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 22:09:02,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:02,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.46 MB 2025-02-14 22:09:02,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15585.47 MB 2025-02-14 22:09:02,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1140.01 MB 2025-02-14 22:09:02,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21099.45 MB 2025-02-14 22:09:02,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21099.45 MB 2025-02-14 22:09:02,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:09:02,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17081.96 MB 2025-02-14 22:09:02,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:09:02,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:09:02,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:09:02,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:02,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16189.09 MB 2025-02-14 22:09:02,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16451.68 MB 2025-02-14 22:09:02,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.58 MB 2025-02-14 22:09:02,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21099.45 MB 2025-02-14 22:09:02,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21269.32 MB 2025-02-14 22:09:02,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-14 22:09:02,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.55 MB 2025-02-14 22:09:02,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:09:02,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:09:02,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:09:02,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:02,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16617.77 MB 2025-02-14 22:09:02,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16846.50 MB 2025-02-14 22:09:02,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-14 22:09:02,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21269.32 MB 2025-02-14 22:09:02,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21269.32 MB 2025-02-14 22:09:02,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:09:02,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16846.50 MB 2025-02-14 22:09:02,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:09:02,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:09:02,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.50 seconds 2025-02-14 22:09:02,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:02,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13348.47 MB 2025-02-14 22:09:02,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17047.20 MB 2025-02-14 22:09:02,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3698.73 MB 2025-02-14 22:09:02,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42777.71 MB 2025-02-14 22:09:02,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21269.32 MB 2025-02-14 22:09:02,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21508.39 MB 2025-02-14 22:09:02,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17047.20 MB 2025-02-14 22:09:02,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:09:02,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:09:02,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 22:09:02,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:02,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17047.20 MB 2025-02-14 22:09:02,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20055.70 MB 2025-02-14 22:09:02,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-14 22:09:02,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21269.32 MB 2025-02-14 22:09:02,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21269.32 MB 2025-02-14 22:09:02,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:09:02,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20356.52 MB 2025-02-14 22:09:02,441 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 22:09:02,441 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 22:09:02,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:09:02,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:09:02,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:09:02,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:09:02,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20055.70 MB 2025-02-14 22:09:02,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28478.91 MB 2025-02-14 22:09:02,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 22:09:02,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21269.32 MB 2025-02-14 22:09:02,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31740.40 MB 2025-02-14 22:09:02,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 22:09:02,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28478.91 MB 2025-02-14 22:09:02,698 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 22:09:02,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:09:02,700 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:09:02,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:09:02,703 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:09:02,710 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:09:02,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:09:02,712 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:09:02,713 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 22:09:49,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:09:49,899 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:09:49,907 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:09:49,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:09:49,915 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1846, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:09:49,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:09:49,917 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1846, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:10:18,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:10:18,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:10:18,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.33 seconds 2025-02-14 22:10:18,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:18,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33996.45 MB 2025-02-14 22:10:18,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40529.34 MB 2025-02-14 22:10:18,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6532.89 MB 2025-02-14 22:10:18,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43285.22 MB 2025-02-14 22:10:18,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47722.79 MB 2025-02-14 22:10:18,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4437.57 MB 2025-02-14 22:10:18,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49357.43 MB 2025-02-14 22:10:18,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:10:18,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:10:18,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:10:18,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:18,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40529.34 MB 2025-02-14 22:10:18,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33539.15 MB 2025-02-14 22:10:18,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6990.19 MB 2025-02-14 22:10:18,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47722.79 MB 2025-02-14 22:10:18,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52242.15 MB 2025-02-14 22:10:18,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4519.36 MB 2025-02-14 22:10:18,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48125.47 MB 2025-02-14 22:10:20,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:10:20,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:10:20,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:10:20,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33539.15 MB 2025-02-14 22:10:20,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34069.99 MB 2025-02-14 22:10:20,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:10:20,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52242.15 MB 2025-02-14 22:10:20,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42603.64 MB 2025-02-14 22:10:20,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9638.51 MB 2025-02-14 22:10:20,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38048.54 MB 2025-02-14 22:10:20,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:10:20,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:10:20,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:10:20,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34069.99 MB 2025-02-14 22:10:20,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35959.52 MB 2025-02-14 22:10:20,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:10:20,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42603.64 MB 2025-02-14 22:10:20,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42605.74 MB 2025-02-14 22:10:20,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 22:10:20,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37376.95 MB 2025-02-14 22:10:20,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:10:20,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:10:20,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:10:20,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35959.52 MB 2025-02-14 22:10:20,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30036.86 MB 2025-02-14 22:10:20,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5922.66 MB 2025-02-14 22:10:20,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42605.74 MB 2025-02-14 22:10:20,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42605.74 MB 2025-02-14 22:10:20,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:10:20,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36687.12 MB 2025-02-14 22:10:20,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:10:20,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:10:20,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:10:20,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34069.99 MB 2025-02-14 22:10:20,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30036.86 MB 2025-02-14 22:10:20,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4033.13 MB 2025-02-14 22:10:20,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42603.64 MB 2025-02-14 22:10:20,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42605.74 MB 2025-02-14 22:10:20,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 22:10:20,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36687.12 MB 2025-02-14 22:10:20,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:10:20,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:10:20,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:10:20,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31570.40 MB 2025-02-14 22:10:20,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32337.40 MB 2025-02-14 22:10:20,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:10:20,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42605.74 MB 2025-02-14 22:10:20,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42855.30 MB 2025-02-14 22:10:20,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 249.56 MB 2025-02-14 22:10:20,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33045.19 MB 2025-02-14 22:10:20,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:10:20,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:10:20,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:10:20,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32750.29 MB 2025-02-14 22:10:20,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32984.62 MB 2025-02-14 22:10:20,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.32 MB 2025-02-14 22:10:20,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42855.30 MB 2025-02-14 22:10:20,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42855.30 MB 2025-02-14 22:10:20,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:10:20,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33167.10 MB 2025-02-14 22:10:20,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:10:20,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:10:20,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.74 seconds 2025-02-14 22:10:20,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27564.84 MB 2025-02-14 22:10:20,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33185.69 MB 2025-02-14 22:10:20,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5620.85 MB 2025-02-14 22:10:20,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40118.52 MB 2025-02-14 22:10:20,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42855.30 MB 2025-02-14 22:10:20,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2736.78 MB 2025-02-14 22:10:20,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33185.69 MB 2025-02-14 22:10:20,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:10:20,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:10:20,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:10:20,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33185.69 MB 2025-02-14 22:10:20,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24404.14 MB 2025-02-14 22:10:20,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8781.55 MB 2025-02-14 22:10:20,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42855.30 MB 2025-02-14 22:10:20,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42855.30 MB 2025-02-14 22:10:20,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:10:20,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35697.36 MB 2025-02-14 22:10:20,950 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:10:20,951 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:10:20,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:10:20,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:10:20,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:10:20,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:10:20,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24404.14 MB 2025-02-14 22:10:20,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32843.17 MB 2025-02-14 22:10:20,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:10:20,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42855.30 MB 2025-02-14 22:10:20,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51246.01 MB 2025-02-14 22:10:20,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:10:20,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32843.17 MB 2025-02-14 22:10:21,116 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:10:21,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:10:21,117 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:10:21,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:10:21,118 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:10:21,123 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:10:21,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:10:21,124 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:10:21,124 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:10:52,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:10:52,468 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:10:52,473 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:10:52,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:10:52,477 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:10:52,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:10:52,478 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:11:10,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:11:10,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:11:10,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.14 seconds 2025-02-14 22:11:10,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:10,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21170.23 MB 2025-02-14 22:11:10,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25335.57 MB 2025-02-14 22:11:10,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4165.34 MB 2025-02-14 22:11:10,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63831.02 MB 2025-02-14 22:11:10,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31939.62 MB 2025-02-14 22:11:10,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31891.39 MB 2025-02-14 22:11:10,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.48 MB 2025-02-14 22:11:10,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:11:10,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:11:10,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:11:10,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:10,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25335.57 MB 2025-02-14 22:11:10,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21896.71 MB 2025-02-14 22:11:10,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3438.86 MB 2025-02-14 22:11:10,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31939.62 MB 2025-02-14 22:11:10,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42205.18 MB 2025-02-14 22:11:10,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10265.56 MB 2025-02-14 22:11:10,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37858.30 MB 2025-02-14 22:11:12,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:11:12,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:11:12,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:11:12,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:12,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21896.71 MB 2025-02-14 22:11:12,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22427.55 MB 2025-02-14 22:11:12,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:11:12,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42205.18 MB 2025-02-14 22:11:12,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25469.91 MB 2025-02-14 22:11:12,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16735.27 MB 2025-02-14 22:11:12,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26406.10 MB 2025-02-14 22:11:12,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:11:12,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:11:12,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:11:12,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:12,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22427.55 MB 2025-02-14 22:11:12,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24317.08 MB 2025-02-14 22:11:12,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:11:12,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25469.91 MB 2025-02-14 22:11:12,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27357.35 MB 2025-02-14 22:11:12,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:11:12,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25734.51 MB 2025-02-14 22:11:12,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:11:12,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:11:12,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:11:12,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:12,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24317.08 MB 2025-02-14 22:11:12,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26558.94 MB 2025-02-14 22:11:12,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:11:12,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27357.35 MB 2025-02-14 22:11:12,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33491.52 MB 2025-02-14 22:11:12,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:11:12,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32103.22 MB 2025-02-14 22:11:12,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:11:12,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:11:12,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:11:12,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:12,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22427.55 MB 2025-02-14 22:11:12,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26558.94 MB 2025-02-14 22:11:12,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:11:12,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25469.91 MB 2025-02-14 22:11:12,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33491.52 MB 2025-02-14 22:11:12,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 22:11:12,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32103.22 MB 2025-02-14 22:11:13,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:11:13,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:11:13,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:11:13,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:13,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28092.48 MB 2025-02-14 22:11:13,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28859.48 MB 2025-02-14 22:11:13,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:11:13,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33491.52 MB 2025-02-14 22:11:13,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33906.75 MB 2025-02-14 22:11:13,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:11:13,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29567.27 MB 2025-02-14 22:11:13,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:11:13,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:11:13,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:11:13,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:13,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29272.37 MB 2025-02-14 22:11:13,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29500.82 MB 2025-02-14 22:11:13,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 22:11:13,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33906.75 MB 2025-02-14 22:11:13,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33906.75 MB 2025-02-14 22:11:13,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:11:13,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29738.51 MB 2025-02-14 22:11:13,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:11:13,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:11:13,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.54 seconds 2025-02-14 22:11:13,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:13,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17069.47 MB 2025-02-14 22:11:13,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29701.30 MB 2025-02-14 22:11:13,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12631.83 MB 2025-02-14 22:11:13,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63831.02 MB 2025-02-14 22:11:13,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33906.75 MB 2025-02-14 22:11:13,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29924.26 MB 2025-02-14 22:11:13,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29738.51 MB 2025-02-14 22:11:13,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:11:13,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:11:13,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:11:13,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:13,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29701.30 MB 2025-02-14 22:11:13,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22062.93 MB 2025-02-14 22:11:13,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7638.37 MB 2025-02-14 22:11:13,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33906.75 MB 2025-02-14 22:11:13,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33906.75 MB 2025-02-14 22:11:13,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:11:13,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32204.06 MB 2025-02-14 22:11:13,309 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 22:11:13,309 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:11:13,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:11:13,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:11:13,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:11:13,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:11:13,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22062.93 MB 2025-02-14 22:11:13,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30472.24 MB 2025-02-14 22:11:13,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 22:11:13,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33906.75 MB 2025-02-14 22:11:13,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42266.00 MB 2025-02-14 22:11:13,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 22:11:13,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30472.24 MB 2025-02-14 22:11:13,470 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 22:11:13,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:11:13,471 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:11:13,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:11:13,472 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:11:13,477 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:11:13,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:11:13,478 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:11:13,478 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:12:15,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:12:15,115 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:12:15,120 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:12:15,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:12:15,124 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 806, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:12:15,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:12:15,125 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 806, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:12:27,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:12:27,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:12:27,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.40 seconds 2025-02-14 22:12:27,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18585.04 MB 2025-02-14 22:12:27,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21437.43 MB 2025-02-14 22:12:27,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2852.39 MB 2025-02-14 22:12:27,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50625.25 MB 2025-02-14 22:12:27,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25149.05 MB 2025-02-14 22:12:27,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25476.20 MB 2025-02-14 22:12:27,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30322.15 MB 2025-02-14 22:12:27,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:12:27,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:12:27,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 22:12:27,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21437.43 MB 2025-02-14 22:12:27,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17587.19 MB 2025-02-14 22:12:27,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3850.24 MB 2025-02-14 22:12:27,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25149.05 MB 2025-02-14 22:12:27,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25149.05 MB 2025-02-14 22:12:27,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:12:27,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22315.47 MB 2025-02-14 22:12:27,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:12:27,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:12:27,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 22:12:27,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17587.19 MB 2025-02-14 22:12:27,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17668.14 MB 2025-02-14 22:12:27,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-14 22:12:27,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25149.05 MB 2025-02-14 22:12:27,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22294.82 MB 2025-02-14 22:12:27,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2854.22 MB 2025-02-14 22:12:27,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21480.44 MB 2025-02-14 22:12:27,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:12:27,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:12:27,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:12:27,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17668.14 MB 2025-02-14 22:12:27,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17956.23 MB 2025-02-14 22:12:27,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-14 22:12:27,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22294.82 MB 2025-02-14 22:12:27,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22294.82 MB 2025-02-14 22:12:27,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:12:27,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18172.39 MB 2025-02-14 22:12:27,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:12:27,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:12:27,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 22:12:27,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17956.23 MB 2025-02-14 22:12:27,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18306.16 MB 2025-02-14 22:12:27,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 349.93 MB 2025-02-14 22:12:27,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22294.82 MB 2025-02-14 22:12:27,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22294.82 MB 2025-02-14 22:12:27,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:12:27,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19144.10 MB 2025-02-14 22:12:27,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:12:27,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:12:27,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:12:27,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17668.14 MB 2025-02-14 22:12:27,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18306.16 MB 2025-02-14 22:12:27,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 638.02 MB 2025-02-14 22:12:27,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22294.82 MB 2025-02-14 22:12:27,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22294.82 MB 2025-02-14 22:12:27,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:12:27,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19144.10 MB 2025-02-14 22:12:27,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:12:27,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:12:27,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 22:12:27,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18645.36 MB 2025-02-14 22:12:27,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18792.31 MB 2025-02-14 22:12:27,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.95 MB 2025-02-14 22:12:27,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22294.82 MB 2025-02-14 22:12:27,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 22:12:27,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 90.18 MB 2025-02-14 22:12:27,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18900.25 MB 2025-02-14 22:12:27,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:12:27,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:12:27,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:12:27,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18885.27 MB 2025-02-14 22:12:27,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19032.03 MB 2025-02-14 22:12:27,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.76 MB 2025-02-14 22:12:27,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22385.00 MB 2025-02-14 22:12:27,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 22:12:27,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:12:27,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19032.03 MB 2025-02-14 22:12:27,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:12:27,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:12:27,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.84 seconds 2025-02-14 22:12:27,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:27,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15776.88 MB 2025-02-14 22:12:27,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19163.87 MB 2025-02-14 22:12:27,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3387.00 MB 2025-02-14 22:12:27,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50625.25 MB 2025-02-14 22:12:27,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 22:12:27,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28240.25 MB 2025-02-14 22:12:27,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19163.87 MB 2025-02-14 22:12:28,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:12:28,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:12:28,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 22:12:28,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:28,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16132.89 MB 2025-02-14 22:12:28,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18109.19 MB 2025-02-14 22:12:28,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1976.30 MB 2025-02-14 22:12:28,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22385.00 MB 2025-02-14 22:12:28,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 22:12:28,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:12:28,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18306.80 MB 2025-02-14 22:12:28,144 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-14 22:12:28,145 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 22:12:28,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:12:28,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:12:28,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:12:28,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:12:28,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18109.19 MB 2025-02-14 22:12:28,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23642.26 MB 2025-02-14 22:12:28,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.07 MB 2025-02-14 22:12:28,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22385.00 MB 2025-02-14 22:12:28,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27887.93 MB 2025-02-14 22:12:28,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5502.93 MB 2025-02-14 22:12:28,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23642.26 MB 2025-02-14 22:12:28,259 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-14 22:12:28,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:12:28,260 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:12:28,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:12:28,261 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:12:28,266 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:12:28,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:12:28,267 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:12:28,267 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 22:14:41,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:14:41,981 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:14:41,987 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:14:41,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:14:41,992 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1264, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:14:41,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:14:41,993 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1264, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:15:01,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:15:01,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:15:01,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.25 seconds 2025-02-14 22:15:01,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:01,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21776.46 MB 2025-02-14 22:15:01,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26249.69 MB 2025-02-14 22:15:01,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4473.23 MB 2025-02-14 22:15:01,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33390.85 MB 2025-02-14 22:15:01,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33590.08 MB 2025-02-14 22:15:01,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 22:15:01,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35098.20 MB 2025-02-14 22:15:01,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:15:01,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:15:01,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:15:01,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:01,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26249.69 MB 2025-02-14 22:15:01,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22348.99 MB 2025-02-14 22:15:01,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3900.69 MB 2025-02-14 22:15:01,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33590.08 MB 2025-02-14 22:15:01,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45787.12 MB 2025-02-14 22:15:01,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12197.04 MB 2025-02-14 22:15:01,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39657.73 MB 2025-02-14 22:15:03,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:15:03,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:15:03,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:15:03,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22348.99 MB 2025-02-14 22:15:03,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.83 MB 2025-02-14 22:15:03,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:15:03,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45787.12 MB 2025-02-14 22:15:03,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27780.97 MB 2025-02-14 22:15:03,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18006.15 MB 2025-02-14 22:15:03,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26858.38 MB 2025-02-14 22:15:03,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:15:03,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:15:03,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:15:03,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22879.83 MB 2025-02-14 22:15:03,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24769.37 MB 2025-02-14 22:15:03,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:15:03,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27780.97 MB 2025-02-14 22:15:03,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28724.69 MB 2025-02-14 22:15:03,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:15:03,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26186.80 MB 2025-02-14 22:15:03,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:15:03,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:15:03,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:15:03,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24769.37 MB 2025-02-14 22:15:03,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27011.22 MB 2025-02-14 22:15:03,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:15:03,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28724.69 MB 2025-02-14 22:15:03,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34389.10 MB 2025-02-14 22:15:03,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 22:15:03,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.51 MB 2025-02-14 22:15:03,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:15:03,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:15:03,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:15:03,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22879.83 MB 2025-02-14 22:15:03,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27011.22 MB 2025-02-14 22:15:03,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:15:03,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27780.97 MB 2025-02-14 22:15:03,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34389.10 MB 2025-02-14 22:15:03,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-14 22:15:03,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.51 MB 2025-02-14 22:15:03,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:15:03,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:15:03,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:15:03,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28544.77 MB 2025-02-14 22:15:03,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29311.77 MB 2025-02-14 22:15:03,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:15:03,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34389.10 MB 2025-02-14 22:15:03,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34806.43 MB 2025-02-14 22:15:03,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:15:03,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30019.56 MB 2025-02-14 22:15:03,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:15:03,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:15:03,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:15:03,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29724.66 MB 2025-02-14 22:15:03,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29952.63 MB 2025-02-14 22:15:03,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 22:15:03,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34806.43 MB 2025-02-14 22:15:03,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34806.43 MB 2025-02-14 22:15:03,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:15:03,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30193.14 MB 2025-02-14 22:15:03,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:15:03,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:15:03,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.66 seconds 2025-02-14 22:15:03,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17372.58 MB 2025-02-14 22:15:03,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30152.53 MB 2025-02-14 22:15:03,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12779.94 MB 2025-02-14 22:15:03,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33390.85 MB 2025-02-14 22:15:03,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34806.43 MB 2025-02-14 22:15:03,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-14 22:15:03,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30193.14 MB 2025-02-14 22:15:03,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:15:03,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:15:03,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:15:03,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30152.53 MB 2025-02-14 22:15:03,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22359.28 MB 2025-02-14 22:15:03,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7793.25 MB 2025-02-14 22:15:03,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34806.43 MB 2025-02-14 22:15:03,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34806.43 MB 2025-02-14 22:15:03,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:15:03,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32649.45 MB 2025-02-14 22:15:03,941 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 22:15:03,942 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:15:03,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:15:03,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:15:03,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:15:03,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:15:03,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22359.28 MB 2025-02-14 22:15:03,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30748.42 MB 2025-02-14 22:15:03,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 22:15:03,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34806.43 MB 2025-02-14 22:15:03,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43148.90 MB 2025-02-14 22:15:03,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 22:15:03,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30748.42 MB 2025-02-14 22:15:04,106 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 22:15:04,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:15:04,107 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:15:04,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:15:04,108 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:15:04,113 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:15:04,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:15:04,114 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:15:04,114 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:15:56,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:15:56,287 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:15:56,292 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:15:56,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:15:56,296 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3297, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:15:56,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:15:56,297 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3297, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:16:47,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:16:47,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:16:47,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 50.74 seconds 2025-02-14 22:16:47,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:47,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35945.10 MB 2025-02-14 22:16:47,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47613.66 MB 2025-02-14 22:16:47,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11668.55 MB 2025-02-14 22:16:47,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74467.77 MB 2025-02-14 22:16:47,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52755.96 MB 2025-02-14 22:16:47,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21711.81 MB 2025-02-14 22:16:47,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59281.56 MB 2025-02-14 22:16:47,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:16:47,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:16:47,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:16:47,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:47,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47613.66 MB 2025-02-14 22:16:47,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32919.97 MB 2025-02-14 22:16:47,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14693.69 MB 2025-02-14 22:16:47,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52755.96 MB 2025-02-14 22:16:47,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 94336.19 MB 2025-02-14 22:16:47,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 41580.23 MB 2025-02-14 22:16:47,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 81196.14 MB 2025-02-14 22:16:49,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:16:49,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:16:49,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.99 seconds 2025-02-14 22:16:49,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32919.97 MB 2025-02-14 22:16:49,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33450.81 MB 2025-02-14 22:16:49,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:16:49,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 94336.19 MB 2025-02-14 22:16:49,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36668.70 MB 2025-02-14 22:16:49,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -57667.49 MB 2025-02-14 22:16:49,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37430.40 MB 2025-02-14 22:16:49,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:16:49,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:16:49,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:16:49,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33450.81 MB 2025-02-14 22:16:49,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35340.35 MB 2025-02-14 22:16:49,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:16:49,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36668.70 MB 2025-02-14 22:16:49,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38558.24 MB 2025-02-14 22:16:49,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-14 22:16:49,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36757.78 MB 2025-02-14 22:16:49,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:16:49,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:16:49,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:16:49,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35340.35 MB 2025-02-14 22:16:49,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37582.20 MB 2025-02-14 22:16:49,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:16:49,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38558.24 MB 2025-02-14 22:16:49,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44692.41 MB 2025-02-14 22:16:49,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:16:49,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43126.48 MB 2025-02-14 22:16:49,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:16:49,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:16:49,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:16:49,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33450.81 MB 2025-02-14 22:16:49,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37582.20 MB 2025-02-14 22:16:49,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:16:49,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36668.70 MB 2025-02-14 22:16:49,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44692.41 MB 2025-02-14 22:16:49,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-14 22:16:49,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43126.48 MB 2025-02-14 22:16:49,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:16:49,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:16:49,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:16:49,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39115.74 MB 2025-02-14 22:16:49,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39882.75 MB 2025-02-14 22:16:49,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:16:49,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44692.41 MB 2025-02-14 22:16:49,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45109.74 MB 2025-02-14 22:16:49,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:16:49,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40590.54 MB 2025-02-14 22:16:49,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:16:49,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:16:49,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:16:49,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40295.64 MB 2025-02-14 22:16:49,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40523.91 MB 2025-02-14 22:16:49,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.27 MB 2025-02-14 22:16:49,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45109.74 MB 2025-02-14 22:16:49,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45109.74 MB 2025-02-14 22:16:49,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:16:49,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40738.11 MB 2025-02-14 22:16:49,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:16:49,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:16:49,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 53.37 seconds 2025-02-14 22:16:49,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24456.90 MB 2025-02-14 22:16:49,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40724.22 MB 2025-02-14 22:16:49,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16267.31 MB 2025-02-14 22:16:49,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62979.57 MB 2025-02-14 22:16:49,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45109.74 MB 2025-02-14 22:16:49,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17869.83 MB 2025-02-14 22:16:49,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40738.11 MB 2025-02-14 22:16:49,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:16:49,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:16:49,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:16:49,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40724.22 MB 2025-02-14 22:16:49,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29449.66 MB 2025-02-14 22:16:49,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11274.56 MB 2025-02-14 22:16:49,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45109.74 MB 2025-02-14 22:16:49,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45109.74 MB 2025-02-14 22:16:49,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:16:49,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43226.53 MB 2025-02-14 22:16:49,960 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 22:16:49,961 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:16:49,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:16:49,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:16:49,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 22:16:49,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:16:49,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29449.66 MB 2025-02-14 22:16:49,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37856.35 MB 2025-02-14 22:16:49,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8406.69 MB 2025-02-14 22:16:49,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45109.74 MB 2025-02-14 22:16:49,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49289.36 MB 2025-02-14 22:16:49,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 22:16:49,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37856.35 MB 2025-02-14 22:16:50,129 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 22:16:50,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:16:50,130 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:16:50,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:16:50,131 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:16:50,136 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:16:50,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:16:50,137 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:16:50,137 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:16:57,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:16:57,599 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:16:57,604 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:16:57,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:16:57,608 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1316, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:16:57,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:16:57,609 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1316, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:17:18,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:17:18,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:17:18,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.54 seconds 2025-02-14 22:17:18,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:18,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22138.81 MB 2025-02-14 22:17:18,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26796.58 MB 2025-02-14 22:17:18,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4657.77 MB 2025-02-14 22:17:18,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57648.61 MB 2025-02-14 22:17:18,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35309.75 MB 2025-02-14 22:17:18,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22338.86 MB 2025-02-14 22:17:18,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35687.04 MB 2025-02-14 22:17:18,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:17:18,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:17:18,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:17:18,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:18,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26796.58 MB 2025-02-14 22:17:18,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22619.32 MB 2025-02-14 22:17:18,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4177.26 MB 2025-02-14 22:17:18,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35309.75 MB 2025-02-14 22:17:18,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40642.81 MB 2025-02-14 22:17:18,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5333.06 MB 2025-02-14 22:17:18,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36267.67 MB 2025-02-14 22:17:20,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:17:20,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:17:20,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 22:17:20,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22619.32 MB 2025-02-14 22:17:20,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23150.17 MB 2025-02-14 22:17:20,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:17:20,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40642.81 MB 2025-02-14 22:17:20,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32067.55 MB 2025-02-14 22:17:20,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8575.25 MB 2025-02-14 22:17:20,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27128.71 MB 2025-02-14 22:17:20,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:17:20,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:17:20,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:17:20,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23150.17 MB 2025-02-14 22:17:20,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25039.70 MB 2025-02-14 22:17:20,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:17:20,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32067.55 MB 2025-02-14 22:17:20,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32067.55 MB 2025-02-14 22:17:20,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:17:20,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26457.13 MB 2025-02-14 22:17:20,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:17:20,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:17:20,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:17:20,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25039.70 MB 2025-02-14 22:17:20,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27281.56 MB 2025-02-14 22:17:20,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:17:20,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32067.55 MB 2025-02-14 22:17:20,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35370.57 MB 2025-02-14 22:17:20,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 22:17:20,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32825.84 MB 2025-02-14 22:17:20,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:17:20,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:17:20,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:17:20,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23150.17 MB 2025-02-14 22:17:20,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27281.56 MB 2025-02-14 22:17:20,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:17:20,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32067.55 MB 2025-02-14 22:17:20,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35370.57 MB 2025-02-14 22:17:20,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 22:17:20,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32825.84 MB 2025-02-14 22:17:20,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:17:20,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:17:20,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:17:20,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28815.10 MB 2025-02-14 22:17:20,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29582.10 MB 2025-02-14 22:17:20,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:17:20,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35370.57 MB 2025-02-14 22:17:20,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35787.90 MB 2025-02-14 22:17:20,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:17:20,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30289.89 MB 2025-02-14 22:17:20,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:17:20,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:17:20,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:17:20,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29994.99 MB 2025-02-14 22:17:20,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30224.56 MB 2025-02-14 22:17:20,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.58 MB 2025-02-14 22:17:20,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35787.90 MB 2025-02-14 22:17:20,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35787.90 MB 2025-02-14 22:17:20,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:17:20,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30434.07 MB 2025-02-14 22:17:20,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:17:20,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:17:20,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.96 seconds 2025-02-14 22:17:20,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17553.76 MB 2025-02-14 22:17:20,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30425.64 MB 2025-02-14 22:17:20,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12871.88 MB 2025-02-14 22:17:20,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57648.61 MB 2025-02-14 22:17:20,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35787.90 MB 2025-02-14 22:17:20,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21860.71 MB 2025-02-14 22:17:20,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30434.07 MB 2025-02-14 22:17:20,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:17:20,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:17:20,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:17:20,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30425.64 MB 2025-02-14 22:17:20,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22558.15 MB 2025-02-14 22:17:20,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7867.49 MB 2025-02-14 22:17:20,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35787.90 MB 2025-02-14 22:17:20,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35787.90 MB 2025-02-14 22:17:20,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:17:20,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32937.30 MB 2025-02-14 22:17:20,856 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:17:20,856 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:17:20,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:17:20,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:17:20,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:17:20,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:17:20,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22558.15 MB 2025-02-14 22:17:20,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30997.17 MB 2025-02-14 22:17:20,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:17:20,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35787.90 MB 2025-02-14 22:17:20,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44178.60 MB 2025-02-14 22:17:20,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:17:20,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30997.17 MB 2025-02-14 22:17:21,018 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:17:21,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:17:21,019 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:17:21,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:17:21,020 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:17:21,025 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:17:21,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:17:21,026 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:17:21,026 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:18:18,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:18:18,175 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:18:18,180 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:18:18,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:18:18,184 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 111, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:18:18,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:18:18,185 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 111, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:18:19,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:18:19,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:18:19,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.73 seconds 2025-02-14 22:18:19,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:19,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13742.17 MB 2025-02-14 22:18:19,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14134.99 MB 2025-02-14 22:18:19,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 392.82 MB 2025-02-14 22:18:19,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56763.61 MB 2025-02-14 22:18:19,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:18:19,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37230.74 MB 2025-02-14 22:18:19,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22987.05 MB 2025-02-14 22:18:19,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:18:19,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:18:19,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:18:19,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:19,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14134.99 MB 2025-02-14 22:18:19,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14325.32 MB 2025-02-14 22:18:19,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 190.32 MB 2025-02-14 22:18:19,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:18:19,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:18:19,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:18:19,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14914.61 MB 2025-02-14 22:18:20,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:18:20,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:18:20,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.53 seconds 2025-02-14 22:18:20,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14325.32 MB 2025-02-14 22:18:20,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14472.63 MB 2025-02-14 22:18:20,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.31 MB 2025-02-14 22:18:20,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:18:20,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:18:20,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:18:20,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18410.03 MB 2025-02-14 22:18:20,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:18:20,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:18:20,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:18:20,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14472.56 MB 2025-02-14 22:18:20,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14996.78 MB 2025-02-14 22:18:20,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 524.22 MB 2025-02-14 22:18:20,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:18:20,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:18:20,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:18:20,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15390.12 MB 2025-02-14 22:18:20,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:18:20,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:18:20,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:18:20,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.78 MB 2025-02-14 22:18:20,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15633.49 MB 2025-02-14 22:18:20,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 636.71 MB 2025-02-14 22:18:20,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:18:20,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:18:20,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:18:20,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17157.43 MB 2025-02-14 22:18:20,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:18:20,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:18:20,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 22:18:20,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14472.56 MB 2025-02-14 22:18:20,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15633.49 MB 2025-02-14 22:18:20,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1160.93 MB 2025-02-14 22:18:20,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:18:20,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:18:20,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:18:20,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17157.43 MB 2025-02-14 22:18:20,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:18:20,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:18:20,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 22:18:20,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16248.18 MB 2025-02-14 22:18:20,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16515.59 MB 2025-02-14 22:18:20,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.40 MB 2025-02-14 22:18:20,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:18:20,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19700.65 MB 2025-02-14 22:18:20,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 22:18:20,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16712.00 MB 2025-02-14 22:18:20,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:18:20,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:18:20,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:18:20,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16684.73 MB 2025-02-14 22:18:20,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16913.75 MB 2025-02-14 22:18:20,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.02 MB 2025-02-14 22:18:20,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19700.65 MB 2025-02-14 22:18:20,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19700.65 MB 2025-02-14 22:18:20,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:18:20,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16913.75 MB 2025-02-14 22:18:20,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:18:20,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:18:20,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.45 seconds 2025-02-14 22:18:20,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13355.44 MB 2025-02-14 22:18:20,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17114.25 MB 2025-02-14 22:18:20,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3758.81 MB 2025-02-14 22:18:20,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56763.61 MB 2025-02-14 22:18:20,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19700.65 MB 2025-02-14 22:18:20,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37062.97 MB 2025-02-14 22:18:20,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17114.25 MB 2025-02-14 22:18:20,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:18:20,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:18:20,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:18:20,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17114.25 MB 2025-02-14 22:18:20,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20119.81 MB 2025-02-14 22:18:20,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-14 22:18:20,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19700.65 MB 2025-02-14 22:18:20,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21177.04 MB 2025-02-14 22:18:20,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1476.40 MB 2025-02-14 22:18:20,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20420.97 MB 2025-02-14 22:18:20,919 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 22:18:20,920 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:18:20,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:18:20,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:18:20,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:18:20,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:18:20,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20119.81 MB 2025-02-14 22:18:20,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28534.76 MB 2025-02-14 22:18:20,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 22:18:20,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21177.04 MB 2025-02-14 22:18:20,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31637.64 MB 2025-02-14 22:18:20,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-14 22:18:20,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28534.76 MB 2025-02-14 22:18:21,087 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 22:18:21,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:18:21,088 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:18:21,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:18:21,089 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:18:21,094 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:18:21,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:18:21,095 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:18:21,095 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:19:12,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:12,742 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:19:12,749 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:19:12,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:12,757 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1150, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:19:12,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:12,759 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1150, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:19:30,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:19:30,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:19:30,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.71 seconds 2025-02-14 22:19:30,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:30,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20982.09 MB 2025-02-14 22:19:30,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25052.66 MB 2025-02-14 22:19:30,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4070.57 MB 2025-02-14 22:19:30,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40007.37 MB 2025-02-14 22:19:30,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28466.74 MB 2025-02-14 22:19:30,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11540.63 MB 2025-02-14 22:19:30,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33850.85 MB 2025-02-14 22:19:30,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:19:30,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:19:30,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:19:30,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:30,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25052.66 MB 2025-02-14 22:19:30,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21757.39 MB 2025-02-14 22:19:30,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3295.27 MB 2025-02-14 22:19:30,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28466.74 MB 2025-02-14 22:19:30,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44983.91 MB 2025-02-14 22:19:30,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16517.17 MB 2025-02-14 22:19:30,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37356.79 MB 2025-02-14 22:19:32,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:19:32,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:19:32,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:19:32,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:32,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21757.39 MB 2025-02-14 22:19:32,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22288.23 MB 2025-02-14 22:19:32,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:19:32,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44983.91 MB 2025-02-14 22:19:32,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26520.58 MB 2025-02-14 22:19:32,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18463.33 MB 2025-02-14 22:19:32,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26267.82 MB 2025-02-14 22:19:32,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:19:32,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:19:32,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:19:32,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:32,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22288.23 MB 2025-02-14 22:19:32,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24177.77 MB 2025-02-14 22:19:32,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:19:32,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26520.58 MB 2025-02-14 22:19:32,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27464.30 MB 2025-02-14 22:19:32,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:19:32,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25595.20 MB 2025-02-14 22:19:32,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:19:32,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:19:32,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:19:32,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:32,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24177.77 MB 2025-02-14 22:19:32,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26419.62 MB 2025-02-14 22:19:32,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:19:32,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27464.30 MB 2025-02-14 22:19:32,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34070.33 MB 2025-02-14 22:19:32,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:19:32,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31963.90 MB 2025-02-14 22:19:32,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:19:32,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:19:32,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:19:32,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:32,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22288.23 MB 2025-02-14 22:19:32,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26419.62 MB 2025-02-14 22:19:32,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:19:32,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26520.58 MB 2025-02-14 22:19:32,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34070.33 MB 2025-02-14 22:19:32,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 22:19:32,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31963.90 MB 2025-02-14 22:19:32,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:19:32,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:19:32,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:19:32,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:32,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27953.16 MB 2025-02-14 22:19:32,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28720.17 MB 2025-02-14 22:19:32,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:19:32,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34070.33 MB 2025-02-14 22:19:32,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34483.47 MB 2025-02-14 22:19:32,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 22:19:32,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29427.96 MB 2025-02-14 22:19:32,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:19:32,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:19:32,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:19:32,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:32,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29133.06 MB 2025-02-14 22:19:32,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29361.40 MB 2025-02-14 22:19:32,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 22:19:32,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34483.47 MB 2025-02-14 22:19:32,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34483.47 MB 2025-02-14 22:19:32,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:19:32,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.44 MB 2025-02-14 22:19:32,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:19:32,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:19:32,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.12 seconds 2025-02-14 22:19:32,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:32,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16975.40 MB 2025-02-14 22:19:32,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29561.66 MB 2025-02-14 22:19:32,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12586.27 MB 2025-02-14 22:19:32,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40007.37 MB 2025-02-14 22:19:32,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34483.47 MB 2025-02-14 22:19:32,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5523.90 MB 2025-02-14 22:19:32,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.44 MB 2025-02-14 22:19:33,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:19:33,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:19:33,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:19:33,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:33,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29561.66 MB 2025-02-14 22:19:33,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21967.44 MB 2025-02-14 22:19:33,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7594.23 MB 2025-02-14 22:19:33,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34483.47 MB 2025-02-14 22:19:33,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34483.47 MB 2025-02-14 22:19:33,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:19:33,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32063.19 MB 2025-02-14 22:19:33,171 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 22:19:33,171 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:19:33,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:19:33,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:19:33,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:19:33,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:19:33,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21967.44 MB 2025-02-14 22:19:33,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30372.55 MB 2025-02-14 22:19:33,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 22:19:33,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34483.47 MB 2025-02-14 22:19:33,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42840.62 MB 2025-02-14 22:19:33,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 22:19:33,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30372.55 MB 2025-02-14 22:19:33,335 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 22:19:33,336 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:33,336 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:19:33,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:33,337 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:19:33,342 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:19:33,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:33,343 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:19:33,343 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:19:43,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:43,231 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:19:43,235 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:19:43,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:43,239 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:19:43,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:19:43,240 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:20:01,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:20:01,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:20:01,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.34 seconds 2025-02-14 22:20:01,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:01,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-14 22:20:01,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-14 22:20:01,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-14 22:20:01,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55375.30 MB 2025-02-14 22:20:01,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30662.46 MB 2025-02-14 22:20:01,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24712.84 MB 2025-02-14 22:20:01,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34308.89 MB 2025-02-14 22:20:01,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:20:01,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:20:01,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:20:01,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:01,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-14 22:20:01,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21928.95 MB 2025-02-14 22:20:01,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3469.66 MB 2025-02-14 22:20:01,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30662.46 MB 2025-02-14 22:20:01,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44312.82 MB 2025-02-14 22:20:01,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13650.36 MB 2025-02-14 22:20:01,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37888.60 MB 2025-02-14 22:20:03,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:20:03,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:20:03,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:20:03,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:03,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.95 MB 2025-02-14 22:20:03,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22459.79 MB 2025-02-14 22:20:03,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:20:03,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44312.82 MB 2025-02-14 22:20:03,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28598.86 MB 2025-02-14 22:20:03,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15713.96 MB 2025-02-14 22:20:03,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26438.34 MB 2025-02-14 22:20:03,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:20:03,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:20:03,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:20:03,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:03,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22459.79 MB 2025-02-14 22:20:03,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24349.32 MB 2025-02-14 22:20:03,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:20:03,595 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 22:20:03,595 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28598.86 MB 2025-02-14 22:20:03,595 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:03,595 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25766.75 MB 2025-02-14 22:20:03,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:20:03,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:20:03,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:20:03,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:03,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24349.32 MB 2025-02-14 22:20:03,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26591.18 MB 2025-02-14 22:20:03,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:20:03,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 22:20:03,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 22:20:03,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:20:03,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32136.23 MB 2025-02-14 22:20:03,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:20:03,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:20:03,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:20:03,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:03,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22459.79 MB 2025-02-14 22:20:03,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26591.18 MB 2025-02-14 22:20:03,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:20:03,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 22:20:03,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 22:20:03,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:20:03,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32136.23 MB 2025-02-14 22:20:03,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:20:03,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:20:03,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:20:03,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:03,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28125.49 MB 2025-02-14 22:20:03,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.49 MB 2025-02-14 22:20:03,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:20:03,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34261.17 MB 2025-02-14 22:20:03,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 22:20:03,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:20:03,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29600.28 MB 2025-02-14 22:20:03,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:20:03,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:20:03,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:20:03,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:03,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29305.38 MB 2025-02-14 22:20:03,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29534.44 MB 2025-02-14 22:20:03,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-14 22:20:03,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 22:20:03,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 22:20:03,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:03,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29773.49 MB 2025-02-14 22:20:03,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:20:03,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:20:03,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.74 seconds 2025-02-14 22:20:03,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:03,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-14 22:20:03,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29735.41 MB 2025-02-14 22:20:03,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12645.04 MB 2025-02-14 22:20:03,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55375.30 MB 2025-02-14 22:20:03,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 22:20:03,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20696.79 MB 2025-02-14 22:20:03,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29773.49 MB 2025-02-14 22:20:04,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:20:04,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:20:04,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:20:04,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:04,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29735.41 MB 2025-02-14 22:20:04,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22094.01 MB 2025-02-14 22:20:04,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7641.41 MB 2025-02-14 22:20:04,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 22:20:04,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 22:20:04,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:04,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32245.85 MB 2025-02-14 22:20:04,270 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 22:20:04,270 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:20:04,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:20:04,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:20:04,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:20:04,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:04,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22094.01 MB 2025-02-14 22:20:04,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30528.86 MB 2025-02-14 22:20:04,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 22:20:04,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 22:20:04,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45162.17 MB 2025-02-14 22:20:04,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10483.66 MB 2025-02-14 22:20:04,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30528.86 MB 2025-02-14 22:20:04,431 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 22:20:04,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:04,432 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:20:04,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:04,433 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:20:04,438 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:20:04,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:04,439 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:20:04,439 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:20:54,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:54,992 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:20:54,997 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:20:55,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:55,000 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:20:55,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:55,001 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:20:58,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:20:58,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:20:58,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-14 22:20:58,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:58,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14438.99 MB 2025-02-14 22:20:58,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15185.71 MB 2025-02-14 22:20:58,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.72 MB 2025-02-14 22:20:58,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57740.89 MB 2025-02-14 22:20:58,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:20:58,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34462.50 MB 2025-02-14 22:20:58,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24136.85 MB 2025-02-14 22:20:58,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:20:58,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:20:58,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:20:58,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:58,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15185.71 MB 2025-02-14 22:20:58,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15266.57 MB 2025-02-14 22:20:58,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.86 MB 2025-02-14 22:20:58,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:20:58,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:20:58,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:58,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17598.26 MB 2025-02-14 22:20:59,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:20:59,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:20:59,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-14 22:20:59,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15266.57 MB 2025-02-14 22:20:59,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15493.50 MB 2025-02-14 22:20:59,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.93 MB 2025-02-14 22:20:59,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:20:59,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:20:59,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:59,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19436.22 MB 2025-02-14 22:20:59,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:20:59,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:20:59,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:20:59,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15493.44 MB 2025-02-14 22:20:59,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16301.02 MB 2025-02-14 22:20:59,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 807.58 MB 2025-02-14 22:20:59,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:20:59,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:20:59,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:59,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16906.97 MB 2025-02-14 22:20:59,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:20:59,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:20:59,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:20:59,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16301.02 MB 2025-02-14 22:20:59,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17259.45 MB 2025-02-14 22:20:59,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 958.43 MB 2025-02-14 22:20:59,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:20:59,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:20:59,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:59,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19629.59 MB 2025-02-14 22:20:59,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:20:59,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:20:59,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:20:59,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15493.44 MB 2025-02-14 22:20:59,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17259.45 MB 2025-02-14 22:20:59,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1766.01 MB 2025-02-14 22:20:59,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:20:59,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:20:59,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:59,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19629.59 MB 2025-02-14 22:20:59,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:20:59,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:20:59,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:20:59,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17915.04 MB 2025-02-14 22:20:59,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18242.93 MB 2025-02-14 22:20:59,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 327.89 MB 2025-02-14 22:20:59,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:20:59,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23454.55 MB 2025-02-14 22:20:59,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 176.16 MB 2025-02-14 22:20:59,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18553.85 MB 2025-02-14 22:20:59,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:20:59,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:20:59,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:20:59,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18419.45 MB 2025-02-14 22:20:59,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18628.85 MB 2025-02-14 22:20:59,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.41 MB 2025-02-14 22:20:59,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23454.55 MB 2025-02-14 22:20:59,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23454.55 MB 2025-02-14 22:20:59,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:59,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18652.12 MB 2025-02-14 22:20:59,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:20:59,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:20:59,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.28 seconds 2025-02-14 22:20:59,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13703.85 MB 2025-02-14 22:20:59,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18829.73 MB 2025-02-14 22:20:59,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5125.88 MB 2025-02-14 22:20:59,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57740.89 MB 2025-02-14 22:20:59,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23454.55 MB 2025-02-14 22:20:59,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34286.34 MB 2025-02-14 22:20:59,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18829.73 MB 2025-02-14 22:20:59,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:20:59,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:20:59,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:20:59,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18829.73 MB 2025-02-14 22:20:59,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17624.47 MB 2025-02-14 22:20:59,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1205.26 MB 2025-02-14 22:20:59,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23454.55 MB 2025-02-14 22:20:59,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23454.55 MB 2025-02-14 22:20:59,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:20:59,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19064.25 MB 2025-02-14 22:20:59,568 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 22:20:59,568 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:20:59,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:20:59,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:20:59,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:20:59,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:20:59,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17624.47 MB 2025-02-14 22:20:59,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26055.15 MB 2025-02-14 22:20:59,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 22:20:59,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23454.55 MB 2025-02-14 22:20:59,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31836.86 MB 2025-02-14 22:20:59,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 22:20:59,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26055.15 MB 2025-02-14 22:20:59,732 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 22:20:59,734 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:59,734 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:20:59,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:59,735 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:20:59,739 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:20:59,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:20:59,740 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:20:59,740 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:21:08,336 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:21:08,336 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:21:08,341 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:21:08,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:21:08,345 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1098, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:21:08,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:21:08,345 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1098, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:21:25,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:21:25,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:21:25,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.97 seconds 2025-02-14 22:21:25,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:25,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20619.75 MB 2025-02-14 22:21:25,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24505.77 MB 2025-02-14 22:21:25,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3886.02 MB 2025-02-14 22:21:25,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44409.29 MB 2025-02-14 22:21:25,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29467.08 MB 2025-02-14 22:21:25,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14942.21 MB 2025-02-14 22:21:25,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33488.50 MB 2025-02-14 22:21:25,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:21:25,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:21:25,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:21:25,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:25,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24505.77 MB 2025-02-14 22:21:25,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21487.06 MB 2025-02-14 22:21:25,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3018.71 MB 2025-02-14 22:21:25,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 22:21:25,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42276.49 MB 2025-02-14 22:21:25,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12809.40 MB 2025-02-14 22:21:25,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36422.74 MB 2025-02-14 22:21:27,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:21:27,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:21:27,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:21:27,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21487.06 MB 2025-02-14 22:21:27,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22017.90 MB 2025-02-14 22:21:27,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:21:27,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42276.49 MB 2025-02-14 22:21:27,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27705.48 MB 2025-02-14 22:21:27,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14571.01 MB 2025-02-14 22:21:27,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25996.45 MB 2025-02-14 22:21:27,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:21:27,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:21:27,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:21:27,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22017.90 MB 2025-02-14 22:21:27,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23907.43 MB 2025-02-14 22:21:27,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:21:27,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27705.48 MB 2025-02-14 22:21:27,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27705.48 MB 2025-02-14 22:21:27,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:21:27,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25324.86 MB 2025-02-14 22:21:27,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:21:27,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:21:27,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:21:27,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23907.43 MB 2025-02-14 22:21:27,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26149.29 MB 2025-02-14 22:21:27,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:21:27,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27705.48 MB 2025-02-14 22:21:27,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33367.79 MB 2025-02-14 22:21:27,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:21:27,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31693.57 MB 2025-02-14 22:21:27,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:21:27,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:21:27,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:21:27,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22017.90 MB 2025-02-14 22:21:27,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26149.29 MB 2025-02-14 22:21:27,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:21:27,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27705.48 MB 2025-02-14 22:21:27,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33367.79 MB 2025-02-14 22:21:27,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:21:27,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31693.57 MB 2025-02-14 22:21:27,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:21:27,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:21:27,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:21:27,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27682.83 MB 2025-02-14 22:21:27,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28449.83 MB 2025-02-14 22:21:27,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:21:27,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33367.79 MB 2025-02-14 22:21:27,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33783.02 MB 2025-02-14 22:21:27,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:21:27,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29157.62 MB 2025-02-14 22:21:27,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:21:27,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:21:27,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:21:27,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28862.72 MB 2025-02-14 22:21:27,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29089.72 MB 2025-02-14 22:21:27,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.99 MB 2025-02-14 22:21:27,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33783.02 MB 2025-02-14 22:21:27,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33783.02 MB 2025-02-14 22:21:27,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:21:27,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29323.65 MB 2025-02-14 22:21:27,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:21:27,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:21:27,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.37 seconds 2025-02-14 22:21:27,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16794.23 MB 2025-02-14 22:21:27,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29290.20 MB 2025-02-14 22:21:27,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12495.97 MB 2025-02-14 22:21:27,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44409.29 MB 2025-02-14 22:21:27,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33783.02 MB 2025-02-14 22:21:27,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10626.27 MB 2025-02-14 22:21:27,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29323.65 MB 2025-02-14 22:21:27,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:21:27,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:21:27,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:21:27,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:27,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29290.20 MB 2025-02-14 22:21:27,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21786.98 MB 2025-02-14 22:21:27,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7503.22 MB 2025-02-14 22:21:27,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33783.02 MB 2025-02-14 22:21:27,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33783.02 MB 2025-02-14 22:21:27,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:21:27,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31792.35 MB 2025-02-14 22:21:28,005 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 22:21:28,005 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:21:28,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:21:28,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:21:28,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:21:28,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:21:28,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21786.98 MB 2025-02-14 22:21:28,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30193.67 MB 2025-02-14 22:21:28,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8406.69 MB 2025-02-14 22:21:28,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33783.02 MB 2025-02-14 22:21:28,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37962.65 MB 2025-02-14 22:21:28,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 22:21:28,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30193.67 MB 2025-02-14 22:21:28,167 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 22:21:28,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:21:28,169 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:21:28,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:21:28,170 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:21:28,174 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:21:28,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:21:28,176 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:21:28,176 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:22:28,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:22:28,487 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:22:28,495 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:22:28,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:22:28,502 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:22:28,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:22:28,504 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:22:30,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:22:30,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:22:30,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.43 seconds 2025-02-14 22:22:30,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:30,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-14 22:22:30,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-14 22:22:30,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 22:22:30,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46321.89 MB 2025-02-14 22:22:30,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 22:22:30,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26319.26 MB 2025-02-14 22:22:30,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23506.21 MB 2025-02-14 22:22:30,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:22:30,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:22:30,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:22:30,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:30,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-14 22:22:30,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14782.45 MB 2025-02-14 22:22:30,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.15 MB 2025-02-14 22:22:30,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 22:22:30,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 22:22:30,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:22:30,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.89 MB 2025-02-14 22:22:31,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:22:31,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:22:31,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 22:22:31,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:31,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14782.45 MB 2025-02-14 22:22:31,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14974.88 MB 2025-02-14 22:22:31,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 22:22:31,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 22:22:31,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:22:31,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 22:22:31,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18953.13 MB 2025-02-14 22:22:31,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:22:31,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:22:31,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:22:31,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:31,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14974.81 MB 2025-02-14 22:22:31,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15659.60 MB 2025-02-14 22:22:31,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 22:22:31,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:22:31,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:22:31,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:22:31,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16173.42 MB 2025-02-14 22:22:31,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:22:31,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:22:31,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:22:31,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:31,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15659.60 MB 2025-02-14 22:22:31,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16472.31 MB 2025-02-14 22:22:31,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 22:22:31,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:22:31,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:22:31,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:22:31,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18482.08 MB 2025-02-14 22:22:31,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:22:31,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:22:31,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:22:31,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:31,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14974.81 MB 2025-02-14 22:22:31,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16472.31 MB 2025-02-14 22:22:31,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 22:22:31,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:22:31,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:22:31,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:22:31,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18482.08 MB 2025-02-14 22:22:31,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:22:31,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:22:31,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 22:22:31,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:31,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17028.22 MB 2025-02-14 22:22:31,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17306.26 MB 2025-02-14 22:22:31,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 22:22:31,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:22:31,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 22:22:31,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 22:22:31,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17572.56 MB 2025-02-14 22:22:31,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:22:31,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:22:31,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:22:31,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:31,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17455.94 MB 2025-02-14 22:22:31,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17679.35 MB 2025-02-14 22:22:31,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.41 MB 2025-02-14 22:22:31,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19679.67 MB 2025-02-14 22:22:31,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 22:22:31,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:22:31,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17689.62 MB 2025-02-14 22:22:31,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:22:31,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:22:31,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.32 seconds 2025-02-14 22:22:31,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:31,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-14 22:22:31,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17880.30 MB 2025-02-14 22:22:31,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4378.53 MB 2025-02-14 22:22:31,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46321.89 MB 2025-02-14 22:22:31,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 22:22:31,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26642.22 MB 2025-02-14 22:22:31,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17880.30 MB 2025-02-14 22:22:32,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:22:32,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:22:32,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:22:32,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:32,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17880.30 MB 2025-02-14 22:22:32,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17300.84 MB 2025-02-14 22:22:32,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -579.47 MB 2025-02-14 22:22:32,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19679.67 MB 2025-02-14 22:22:32,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19679.67 MB 2025-02-14 22:22:32,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:22:32,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18984.76 MB 2025-02-14 22:22:32,113 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 22:22:32,113 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:22:32,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:22:32,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:22:32,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:22:32,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:22:32,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17300.84 MB 2025-02-14 22:22:32,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25735.45 MB 2025-02-14 22:22:32,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 22:22:32,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19679.67 MB 2025-02-14 22:22:32,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30161.24 MB 2025-02-14 22:22:32,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 22:22:32,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25735.45 MB 2025-02-14 22:22:32,275 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 22:22:32,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:22:32,280 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:22:32,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:22:32,281 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:22:32,285 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:22:32,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:22:32,286 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:22:32,286 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:23:58,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:23:58,248 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:23:58,253 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:23:58,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:23:58,258 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1199, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:23:58,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:23:58,259 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1199, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:24:16,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:24:16,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:24:16,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.30 seconds 2025-02-14 22:24:16,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:16,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21323.53 MB 2025-02-14 22:24:16,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25566.72 MB 2025-02-14 22:24:16,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4243.19 MB 2025-02-14 22:24:16,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38545.65 MB 2025-02-14 22:24:16,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28649.19 MB 2025-02-14 22:24:16,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9896.46 MB 2025-02-14 22:24:16,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34418.78 MB 2025-02-14 22:24:16,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:24:16,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:24:16,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:24:16,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:16,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25566.72 MB 2025-02-14 22:24:16,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22012.13 MB 2025-02-14 22:24:16,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3554.60 MB 2025-02-14 22:24:16,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28649.19 MB 2025-02-14 22:24:16,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44795.17 MB 2025-02-14 22:24:16,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16145.97 MB 2025-02-14 22:24:16,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37515.71 MB 2025-02-14 22:24:18,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:24:18,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:24:18,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:24:18,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:18,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22012.13 MB 2025-02-14 22:24:18,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22542.97 MB 2025-02-14 22:24:18,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:24:18,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44795.17 MB 2025-02-14 22:24:18,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26528.97 MB 2025-02-14 22:24:18,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18266.19 MB 2025-02-14 22:24:18,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26522.55 MB 2025-02-14 22:24:18,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:24:18,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:24:18,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:24:18,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:18,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22542.97 MB 2025-02-14 22:24:18,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24432.50 MB 2025-02-14 22:24:18,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:24:18,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26528.97 MB 2025-02-14 22:24:18,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27472.69 MB 2025-02-14 22:24:18,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:24:18,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25849.93 MB 2025-02-14 22:24:18,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:24:18,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:24:18,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:24:18,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:18,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24432.50 MB 2025-02-14 22:24:18,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26674.36 MB 2025-02-14 22:24:18,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:24:18,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27472.69 MB 2025-02-14 22:24:18,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34078.72 MB 2025-02-14 22:24:18,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:24:18,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32218.64 MB 2025-02-14 22:24:18,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:24:18,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:24:18,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:24:18,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:18,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22542.97 MB 2025-02-14 22:24:18,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26674.36 MB 2025-02-14 22:24:18,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:24:18,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26528.97 MB 2025-02-14 22:24:18,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34078.72 MB 2025-02-14 22:24:18,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 22:24:18,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32218.64 MB 2025-02-14 22:24:18,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:24:18,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:24:18,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:24:18,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:18,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28207.90 MB 2025-02-14 22:24:18,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28974.90 MB 2025-02-14 22:24:18,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:24:18,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34078.72 MB 2025-02-14 22:24:18,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 22:24:18,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:24:18,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29682.69 MB 2025-02-14 22:24:18,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:24:18,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:24:18,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:24:18,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:18,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29387.79 MB 2025-02-14 22:24:18,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29616.48 MB 2025-02-14 22:24:18,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-14 22:24:18,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 22:24:18,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 22:24:18,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:24:18,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29820.81 MB 2025-02-14 22:24:18,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:24:18,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:24:18,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.72 seconds 2025-02-14 22:24:18,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:18,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17146.12 MB 2025-02-14 22:24:18,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29817.09 MB 2025-02-14 22:24:18,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12670.97 MB 2025-02-14 22:24:18,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38545.65 MB 2025-02-14 22:24:18,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 22:24:18,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4051.70 MB 2025-02-14 22:24:18,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29820.81 MB 2025-02-14 22:24:19,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:24:19,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:24:19,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:24:19,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:19,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29817.09 MB 2025-02-14 22:24:19,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22143.27 MB 2025-02-14 22:24:19,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7673.82 MB 2025-02-14 22:24:19,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 22:24:19,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 22:24:19,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:24:19,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32322.92 MB 2025-02-14 22:24:19,264 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 22:24:19,264 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:24:19,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:24:19,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:24:19,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:24:19,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:19,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22143.27 MB 2025-02-14 22:24:19,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30562.35 MB 2025-02-14 22:24:19,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 22:24:19,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 22:24:19,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42865.79 MB 2025-02-14 22:24:19,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 22:24:19,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30562.35 MB 2025-02-14 22:24:19,433 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 22:24:19,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:24:19,435 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:24:19,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:24:19,436 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:24:19,441 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:24:19,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:24:19,442 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:24:19,442 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:24:28,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:24:28,269 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:24:28,273 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:24:28,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:24:28,277 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1955, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:24:28,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:24:28,278 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1955, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:24:58,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:24:58,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:24:58,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.37 seconds 2025-02-14 22:24:58,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:58,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26591.46 MB 2025-02-14 22:24:58,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33510.09 MB 2025-02-14 22:24:58,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6918.64 MB 2025-02-14 22:24:58,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51237.62 MB 2025-02-14 22:24:58,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37593.55 MB 2025-02-14 22:24:58,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13644.07 MB 2025-02-14 22:24:58,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42405.42 MB 2025-02-14 22:24:58,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:24:58,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:24:58,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 22:24:58,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:24:58,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33510.09 MB 2025-02-14 22:24:58,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25941.29 MB 2025-02-14 22:24:58,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7568.81 MB 2025-02-14 22:24:58,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37593.55 MB 2025-02-14 22:24:58,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61815.65 MB 2025-02-14 22:24:58,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24222.11 MB 2025-02-14 22:24:58,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52389.78 MB 2025-02-14 22:25:00,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:25:00,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:25:00,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 22:25:00,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:00,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25941.29 MB 2025-02-14 22:25:00,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26472.13 MB 2025-02-14 22:25:00,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:25:00,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61815.65 MB 2025-02-14 22:25:00,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-14 22:25:00,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29727.13 MB 2025-02-14 22:25:00,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30450.67 MB 2025-02-14 22:25:00,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:25:00,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:25:00,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:25:00,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:00,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26472.13 MB 2025-02-14 22:25:00,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28361.66 MB 2025-02-14 22:25:00,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:25:00,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 22:25:00,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-14 22:25:00,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:00,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.09 MB 2025-02-14 22:25:00,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:25:00,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:25:00,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:25:00,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:00,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28361.66 MB 2025-02-14 22:25:00,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30603.52 MB 2025-02-14 22:25:00,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:25:00,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 22:25:00,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37750.83 MB 2025-02-14 22:25:00,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:25:00,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36147.80 MB 2025-02-14 22:25:00,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:25:00,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:25:00,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:25:00,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:00,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26472.13 MB 2025-02-14 22:25:00,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30603.52 MB 2025-02-14 22:25:00,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:25:00,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 22:25:00,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37750.83 MB 2025-02-14 22:25:00,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:25:00,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36147.80 MB 2025-02-14 22:25:01,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:25:01,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:25:01,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:25:01,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:01,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32137.06 MB 2025-02-14 22:25:01,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32904.06 MB 2025-02-14 22:25:01,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:25:01,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37750.83 MB 2025-02-14 22:25:01,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 22:25:01,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:25:01,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33611.85 MB 2025-02-14 22:25:01,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:25:01,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:25:01,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:25:01,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:01,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33316.95 MB 2025-02-14 22:25:01,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33545.04 MB 2025-02-14 22:25:01,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.09 MB 2025-02-14 22:25:01,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38166.07 MB 2025-02-14 22:25:01,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 22:25:01,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:01,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33745.50 MB 2025-02-14 22:25:01,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:25:01,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:25:01,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.86 seconds 2025-02-14 22:25:01,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:01,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19780.08 MB 2025-02-14 22:25:01,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33745.52 MB 2025-02-14 22:25:01,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13965.44 MB 2025-02-14 22:25:01,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51237.62 MB 2025-02-14 22:25:01,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 22:25:01,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13071.55 MB 2025-02-14 22:25:01,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33745.52 MB 2025-02-14 22:25:01,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:25:01,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:25:01,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:25:01,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:01,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33745.52 MB 2025-02-14 22:25:01,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24771.41 MB 2025-02-14 22:25:01,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8974.11 MB 2025-02-14 22:25:01,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38166.07 MB 2025-02-14 22:25:01,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38166.07 MB 2025-02-14 22:25:01,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:01,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36246.44 MB 2025-02-14 22:25:01,425 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 22:25:01,425 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:25:01,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:25:01,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:25:01,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:25:01,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:01,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24771.41 MB 2025-02-14 22:25:01,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33173.97 MB 2025-02-14 22:25:01,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8402.56 MB 2025-02-14 22:25:01,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38166.07 MB 2025-02-14 22:25:01,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42343.60 MB 2025-02-14 22:25:01,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 22:25:01,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33173.97 MB 2025-02-14 22:25:01,587 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 22:25:01,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:01,588 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:25:01,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:01,589 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:25:01,594 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:25:01,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:01,595 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:25:01,595 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:25:09,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:09,915 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:25:09,921 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:25:09,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:09,924 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:25:09,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:09,925 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:25:12,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:25:12,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:25:12,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.52 seconds 2025-02-14 22:25:12,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:12,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 22:25:12,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 22:25:12,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 22:25:12,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50698.65 MB 2025-02-14 22:25:12,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20948.45 MB 2025-02-14 22:25:12,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29750.20 MB 2025-02-14 22:25:12,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-14 22:25:12,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:25:12,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:25:12,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:25:12,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:12,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 22:25:12,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-14 22:25:12,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-14 22:25:12,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20948.45 MB 2025-02-14 22:25:12,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20948.45 MB 2025-02-14 22:25:12,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:12,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.37 MB 2025-02-14 22:25:13,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:25:13,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:25:13,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 22:25:13,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-14 22:25:13,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-14 22:25:13,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 22:25:13,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20948.45 MB 2025-02-14 22:25:13,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:25:13,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1415.58 MB 2025-02-14 22:25:13,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19107.09 MB 2025-02-14 22:25:13,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:25:13,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:25:13,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:25:13,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 22:25:13,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-14 22:25:13,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 22:25:13,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:25:13,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 22:25:13,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:13,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-14 22:25:13,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:25:13,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:25:13,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:25:13,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-14 22:25:13,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 22:25:13,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 22:25:13,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:25:13,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20296.24 MB 2025-02-14 22:25:13,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 763.36 MB 2025-02-14 22:25:13,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 22:25:13,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:25:13,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:25:13,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:25:13,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 22:25:13,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 22:25:13,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 22:25:13,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 22:25:13,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20296.24 MB 2025-02-14 22:25:13,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 763.36 MB 2025-02-14 22:25:13,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 22:25:13,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:25:13,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:25:13,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 22:25:13,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-14 22:25:13,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.71 MB 2025-02-14 22:25:13,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 22:25:13,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20296.24 MB 2025-02-14 22:25:13,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:25:13,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 22:25:13,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18031.90 MB 2025-02-14 22:25:13,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:25:13,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:25:13,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:25:13,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.90 MB 2025-02-14 22:25:13,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18133.78 MB 2025-02-14 22:25:13,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.88 MB 2025-02-14 22:25:13,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20459.81 MB 2025-02-14 22:25:13,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:25:13,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:13,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18149.55 MB 2025-02-14 22:25:13,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:25:13,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:25:13,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.49 seconds 2025-02-14 22:25:13,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 22:25:13,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18334.61 MB 2025-02-14 22:25:13,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4804.97 MB 2025-02-14 22:25:13,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50698.65 MB 2025-02-14 22:25:13,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:25:13,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30238.83 MB 2025-02-14 22:25:13,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18334.61 MB 2025-02-14 22:25:13,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:25:13,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:25:13,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:25:13,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18334.61 MB 2025-02-14 22:25:13,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17402.31 MB 2025-02-14 22:25:13,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -932.30 MB 2025-02-14 22:25:13,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20459.81 MB 2025-02-14 22:25:13,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:25:13,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:13,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19137.36 MB 2025-02-14 22:25:13,701 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 22:25:13,701 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 22:25:13,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:25:13,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:25:13,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:25:13,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:13,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17402.31 MB 2025-02-14 22:25:13,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25831.44 MB 2025-02-14 22:25:13,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 22:25:13,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20459.81 MB 2025-02-14 22:25:13,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30935.09 MB 2025-02-14 22:25:13,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 22:25:13,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25831.44 MB 2025-02-14 22:25:13,862 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 22:25:13,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:13,864 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:25:13,865 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:13,865 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:25:13,869 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:25:13,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:13,870 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:25:13,871 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 22:25:20,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:20,341 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:25:20,349 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:25:20,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:20,355 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:25:20,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:20,357 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:25:22,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:25:22,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:25:22,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.68 seconds 2025-02-14 22:25:22,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13714.30 MB 2025-02-14 22:25:22,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14092.97 MB 2025-02-14 22:25:22,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 378.67 MB 2025-02-14 22:25:22,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39315.31 MB 2025-02-14 22:25:22,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20786.97 MB 2025-02-14 22:25:22,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18528.34 MB 2025-02-14 22:25:22,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22959.18 MB 2025-02-14 22:25:22,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:25:22,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:25:22,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:25:22,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14092.97 MB 2025-02-14 22:25:22,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14276.43 MB 2025-02-14 22:25:22,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 183.46 MB 2025-02-14 22:25:22,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20786.97 MB 2025-02-14 22:25:22,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20786.97 MB 2025-02-14 22:25:22,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:22,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.49 MB 2025-02-14 22:25:22,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:25:22,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:25:22,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.51 seconds 2025-02-14 22:25:22,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14276.43 MB 2025-02-14 22:25:22,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14418.43 MB 2025-02-14 22:25:22,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 142.00 MB 2025-02-14 22:25:22,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20786.97 MB 2025-02-14 22:25:22,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20786.97 MB 2025-02-14 22:25:22,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:22,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18361.15 MB 2025-02-14 22:25:22,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:25:22,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:25:22,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:25:22,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14418.36 MB 2025-02-14 22:25:22,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.69 MB 2025-02-14 22:25:22,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 505.33 MB 2025-02-14 22:25:22,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20786.97 MB 2025-02-14 22:25:22,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20786.97 MB 2025-02-14 22:25:22,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:22,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15302.86 MB 2025-02-14 22:25:22,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:25:22,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:25:22,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:25:22,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.69 MB 2025-02-14 22:25:22,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15537.46 MB 2025-02-14 22:25:22,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.77 MB 2025-02-14 22:25:22,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20786.97 MB 2025-02-14 22:25:22,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20786.97 MB 2025-02-14 22:25:22,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:22,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17006.48 MB 2025-02-14 22:25:22,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:25:22,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:25:22,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:25:22,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14418.36 MB 2025-02-14 22:25:22,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15537.46 MB 2025-02-14 22:25:22,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1119.10 MB 2025-02-14 22:25:22,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20786.97 MB 2025-02-14 22:25:22,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20786.97 MB 2025-02-14 22:25:22,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:22,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17006.48 MB 2025-02-14 22:25:22,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:25:22,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:25:22,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 22:25:22,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16130.00 MB 2025-02-14 22:25:22,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16387.77 MB 2025-02-14 22:25:22,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 257.77 MB 2025-02-14 22:25:22,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20786.97 MB 2025-02-14 22:25:22,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-14 22:25:22,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 22:25:22,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16577.10 MB 2025-02-14 22:25:22,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:25:22,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:25:22,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:25:22,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16550.82 MB 2025-02-14 22:25:22,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16779.25 MB 2025-02-14 22:25:22,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.43 MB 2025-02-14 22:25:22,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20950.55 MB 2025-02-14 22:25:22,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-14 22:25:22,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:22,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16779.25 MB 2025-02-14 22:25:22,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:25:22,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:25:22,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.38 seconds 2025-02-14 22:25:22,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:22,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13341.50 MB 2025-02-14 22:25:22,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16980.15 MB 2025-02-14 22:25:22,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3638.64 MB 2025-02-14 22:25:22,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39315.31 MB 2025-02-14 22:25:22,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-14 22:25:22,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18364.76 MB 2025-02-14 22:25:22,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16980.15 MB 2025-02-14 22:25:23,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:25:23,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:25:23,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:25:23,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:23,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14047.18 MB 2025-02-14 22:25:23,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17058.63 MB 2025-02-14 22:25:23,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.45 MB 2025-02-14 22:25:23,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20950.55 MB 2025-02-14 22:25:23,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20950.55 MB 2025-02-14 22:25:23,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:25:23,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17359.74 MB 2025-02-14 22:25:23,020 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 22:25:23,020 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:25:23,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:25:23,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:25:23,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:25:23,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:25:23,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17058.63 MB 2025-02-14 22:25:23,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25490.10 MB 2025-02-14 22:25:23,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 22:25:23,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20950.55 MB 2025-02-14 22:25:23,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29334.96 MB 2025-02-14 22:25:23,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 22:25:23,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25490.10 MB 2025-02-14 22:25:23,184 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 22:25:23,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:23,186 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:25:23,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:23,187 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:25:23,191 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:25:23,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:25:23,193 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:25:23,193 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:26:30,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:26:30,204 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:26:30,209 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:26:30,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:26:30,212 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 93, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:26:30,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:26:30,213 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 93, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:26:31,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:26:31,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:26:31,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.43 seconds 2025-02-14 22:26:31,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:31,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18791.41 MB 2025-02-14 22:26:31,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19120.53 MB 2025-02-14 22:26:31,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 329.12 MB 2025-02-14 22:26:31,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37719.38 MB 2025-02-14 22:26:31,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24983.37 MB 2025-02-14 22:26:31,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12736.00 MB 2025-02-14 22:26:31,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28036.29 MB 2025-02-14 22:26:31,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:26:31,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:26:31,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:26:31,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:31,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19120.53 MB 2025-02-14 22:26:31,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19279.99 MB 2025-02-14 22:26:31,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 159.46 MB 2025-02-14 22:26:31,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24983.37 MB 2025-02-14 22:26:31,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24983.37 MB 2025-02-14 22:26:31,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:26:31,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19773.74 MB 2025-02-14 22:26:32,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:26:32,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:26:32,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.45 seconds 2025-02-14 22:26:32,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19279.99 MB 2025-02-14 22:26:32,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19403.41 MB 2025-02-14 22:26:32,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.42 MB 2025-02-14 22:26:32,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24983.37 MB 2025-02-14 22:26:32,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24983.37 MB 2025-02-14 22:26:32,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:26:32,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23364.70 MB 2025-02-14 22:26:32,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:26:32,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:26:32,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:26:32,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14228.68 MB 2025-02-14 22:26:32,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14667.89 MB 2025-02-14 22:26:32,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.21 MB 2025-02-14 22:26:32,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24983.37 MB 2025-02-14 22:26:32,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24983.37 MB 2025-02-14 22:26:32,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:26:32,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14997.45 MB 2025-02-14 22:26:32,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:26:32,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:26:32,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:26:32,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14667.89 MB 2025-02-14 22:26:32,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15201.36 MB 2025-02-14 22:26:32,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 533.47 MB 2025-02-14 22:26:32,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24983.37 MB 2025-02-14 22:26:32,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24983.37 MB 2025-02-14 22:26:32,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:26:32,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16478.17 MB 2025-02-14 22:26:32,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:26:32,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:26:32,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:26:32,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14228.68 MB 2025-02-14 22:26:32,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15201.36 MB 2025-02-14 22:26:32,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 972.68 MB 2025-02-14 22:26:32,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24983.37 MB 2025-02-14 22:26:32,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24983.37 MB 2025-02-14 22:26:32,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:26:32,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16478.17 MB 2025-02-14 22:26:32,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:26:32,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:26:32,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 22:26:32,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15716.38 MB 2025-02-14 22:26:32,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15940.42 MB 2025-02-14 22:26:32,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.04 MB 2025-02-14 22:26:32,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24983.37 MB 2025-02-14 22:26:32,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 22:26:32,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-14 22:26:32,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16104.98 MB 2025-02-14 22:26:32,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:26:32,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:26:32,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:26:32,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16082.13 MB 2025-02-14 22:26:32,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16304.17 MB 2025-02-14 22:26:32,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.03 MB 2025-02-14 22:26:32,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 22:26:32,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 22:26:32,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:26:32,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16304.17 MB 2025-02-14 22:26:32,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:26:32,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:26:32,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.04 seconds 2025-02-14 22:26:32,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18467.39 MB 2025-02-14 22:26:32,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16502.12 MB 2025-02-14 22:26:32,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1965.27 MB 2025-02-14 22:26:32,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37719.38 MB 2025-02-14 22:26:32,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 22:26:32,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12599.69 MB 2025-02-14 22:26:32,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16502.12 MB 2025-02-14 22:26:32,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:26:32,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:26:32,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:26:32,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13832.68 MB 2025-02-14 22:26:32,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16799.90 MB 2025-02-14 22:26:32,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2967.22 MB 2025-02-14 22:26:32,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 22:26:32,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 22:26:32,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:26:32,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17096.59 MB 2025-02-14 22:26:32,541 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8035, cut from 8037 2025-02-14 22:26:32,541 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:26:32,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:26:32,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:26:32,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:26:32,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:26:32,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16799.90 MB 2025-02-14 22:26:32,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25107.98 MB 2025-02-14 22:26:32,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8308.08 MB 2025-02-14 22:26:32,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 22:26:32,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29248.98 MB 2025-02-14 22:26:32,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4129.29 MB 2025-02-14 22:26:32,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25107.98 MB 2025-02-14 22:26:32,701 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7827] 2025-02-14 22:26:32,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:26:32,702 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:26:32,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:26:32,703 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:26:32,708 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:26:32,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:26:32,709 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:26:32,709 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:27:50,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:27:50,741 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:27:50,746 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:27:50,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:27:50,751 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1386, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:27:50,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:27:50,752 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1386, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:28:11,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:28:11,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:28:11,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.18 seconds 2025-02-14 22:28:11,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:11,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22626.58 MB 2025-02-14 22:28:11,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27531.81 MB 2025-02-14 22:28:11,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4905.24 MB 2025-02-14 22:28:11,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41638.95 MB 2025-02-14 22:28:11,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35406.22 MB 2025-02-14 22:28:11,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6232.74 MB 2025-02-14 22:28:11,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36401.30 MB 2025-02-14 22:28:12,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:28:12,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:28:12,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:28:12,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:12,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27531.81 MB 2025-02-14 22:28:12,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22983.23 MB 2025-02-14 22:28:12,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4548.58 MB 2025-02-14 22:28:12,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35406.22 MB 2025-02-14 22:28:12,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45973.77 MB 2025-02-14 22:28:12,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10567.55 MB 2025-02-14 22:28:12,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41287.89 MB 2025-02-14 22:28:13,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:28:13,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:28:13,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:28:13,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:13,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22983.23 MB 2025-02-14 22:28:13,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23514.07 MB 2025-02-14 22:28:13,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:28:13,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45973.77 MB 2025-02-14 22:28:13,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26371.69 MB 2025-02-14 22:28:13,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19602.08 MB 2025-02-14 22:28:13,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27492.62 MB 2025-02-14 22:28:13,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:28:13,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:28:13,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:28:13,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:13,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23514.07 MB 2025-02-14 22:28:13,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25403.61 MB 2025-02-14 22:28:13,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:28:13,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26371.69 MB 2025-02-14 22:28:13,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28259.12 MB 2025-02-14 22:28:13,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:28:13,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26821.04 MB 2025-02-14 22:28:14,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:28:14,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:28:14,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:28:14,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:14,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25403.61 MB 2025-02-14 22:28:14,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27645.46 MB 2025-02-14 22:28:14,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:28:14,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28259.12 MB 2025-02-14 22:28:14,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35337.01 MB 2025-02-14 22:28:14,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 22:28:14,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33189.75 MB 2025-02-14 22:28:14,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:28:14,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:28:14,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:28:14,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:14,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23514.07 MB 2025-02-14 22:28:14,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27645.46 MB 2025-02-14 22:28:14,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:28:14,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26371.69 MB 2025-02-14 22:28:14,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35337.01 MB 2025-02-14 22:28:14,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 22:28:14,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33189.75 MB 2025-02-14 22:28:14,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:28:14,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:28:14,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:28:14,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:14,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29179.01 MB 2025-02-14 22:28:14,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29946.01 MB 2025-02-14 22:28:14,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:28:14,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35337.01 MB 2025-02-14 22:28:14,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35754.34 MB 2025-02-14 22:28:14,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:28:14,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30653.80 MB 2025-02-14 22:28:14,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:28:14,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:28:14,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:28:14,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:14,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30358.90 MB 2025-02-14 22:28:14,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30587.00 MB 2025-02-14 22:28:14,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 22:28:14,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35754.34 MB 2025-02-14 22:28:14,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35754.34 MB 2025-02-14 22:28:14,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:28:14,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30800.50 MB 2025-02-14 22:28:14,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:28:14,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:28:14,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.58 seconds 2025-02-14 22:28:14,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:14,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17797.64 MB 2025-02-14 22:28:14,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30787.01 MB 2025-02-14 22:28:14,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12989.37 MB 2025-02-14 22:28:14,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41638.95 MB 2025-02-14 22:28:14,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35754.34 MB 2025-02-14 22:28:14,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5884.61 MB 2025-02-14 22:28:14,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30800.50 MB 2025-02-14 22:28:14,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:28:14,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:28:14,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:28:14,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:14,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30787.01 MB 2025-02-14 22:28:14,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22786.12 MB 2025-02-14 22:28:14,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8000.90 MB 2025-02-14 22:28:14,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35754.34 MB 2025-02-14 22:28:14,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35754.34 MB 2025-02-14 22:28:14,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:28:14,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33285.47 MB 2025-02-14 22:28:14,619 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 22:28:14,619 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:28:14,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:28:14,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:28:14,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:28:14,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:28:14,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22786.12 MB 2025-02-14 22:28:14,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31181.33 MB 2025-02-14 22:28:14,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 22:28:14,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35754.34 MB 2025-02-14 22:28:14,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44101.01 MB 2025-02-14 22:28:14,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 22:28:14,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31181.33 MB 2025-02-14 22:28:14,781 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 22:28:14,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:28:14,782 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:28:14,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:28:14,783 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:28:14,788 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:28:14,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:28:14,789 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:28:14,789 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:28:56,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:28:56,875 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:28:56,880 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:28:56,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:28:56,884 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1931, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:28:56,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:28:56,885 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1931, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:29:26,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:29:26,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:29:26,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.83 seconds 2025-02-14 22:29:26,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:26,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26424.22 MB 2025-02-14 22:29:26,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33258.84 MB 2025-02-14 22:29:26,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6834.62 MB 2025-02-14 22:29:26,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52447.67 MB 2025-02-14 22:29:26,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37379.64 MB 2025-02-14 22:29:26,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15068.04 MB 2025-02-14 22:29:26,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42238.19 MB 2025-02-14 22:29:26,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:29:26,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:29:26,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 22:29:26,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:26,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33258.84 MB 2025-02-14 22:29:26,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25816.52 MB 2025-02-14 22:29:26,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7442.32 MB 2025-02-14 22:29:26,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37379.64 MB 2025-02-14 22:29:26,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62448.99 MB 2025-02-14 22:29:26,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25069.36 MB 2025-02-14 22:29:26,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52798.07 MB 2025-02-14 22:29:28,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:29:28,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:29:28,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 22:29:28,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:28,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25816.52 MB 2025-02-14 22:29:28,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26347.36 MB 2025-02-14 22:29:28,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:29:28,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62448.99 MB 2025-02-14 22:29:28,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31960.60 MB 2025-02-14 22:29:28,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30488.40 MB 2025-02-14 22:29:28,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30325.91 MB 2025-02-14 22:29:28,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:29:28,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:29:28,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:29:28,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:28,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-14 22:29:28,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28236.89 MB 2025-02-14 22:29:28,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:29:28,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31960.60 MB 2025-02-14 22:29:28,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31960.60 MB 2025-02-14 22:29:28,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:29:28,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29654.32 MB 2025-02-14 22:29:29,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:29:29,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:29:29,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:29:29,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:29,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28236.89 MB 2025-02-14 22:29:29,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-14 22:29:29,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:29:29,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31960.60 MB 2025-02-14 22:29:29,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37622.91 MB 2025-02-14 22:29:29,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:29:29,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-14 22:29:29,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:29:29,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:29:29,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:29:29,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:29,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-14 22:29:29,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-14 22:29:29,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:29:29,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31960.60 MB 2025-02-14 22:29:29,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37622.91 MB 2025-02-14 22:29:29,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:29:29,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-14 22:29:29,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:29:29,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:29:29,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:29:29,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:29,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32012.29 MB 2025-02-14 22:29:29,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32779.29 MB 2025-02-14 22:29:29,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:29:29,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37622.91 MB 2025-02-14 22:29:29,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38040.24 MB 2025-02-14 22:29:29,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:29:29,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.08 MB 2025-02-14 22:29:29,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:29:29,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:29:29,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:29:29,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:29,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33192.18 MB 2025-02-14 22:29:29,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33421.27 MB 2025-02-14 22:29:29,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-14 22:29:29,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38040.24 MB 2025-02-14 22:29:29,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38040.24 MB 2025-02-14 22:29:29,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:29:29,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33644.53 MB 2025-02-14 22:29:29,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:29:29,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:29:29,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.31 seconds 2025-02-14 22:29:29,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:29,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19696.46 MB 2025-02-14 22:29:29,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33622.26 MB 2025-02-14 22:29:29,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13925.80 MB 2025-02-14 22:29:29,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52447.67 MB 2025-02-14 22:29:29,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38040.24 MB 2025-02-14 22:29:29,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14407.43 MB 2025-02-14 22:29:29,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33644.53 MB 2025-02-14 22:29:29,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:29:29,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:29:29,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:29:29,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:29,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33622.26 MB 2025-02-14 22:29:29,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24699.71 MB 2025-02-14 22:29:29,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8922.55 MB 2025-02-14 22:29:29,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38040.24 MB 2025-02-14 22:29:29,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38040.24 MB 2025-02-14 22:29:29,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:29:29,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36133.01 MB 2025-02-14 22:29:29,484 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 22:29:29,485 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:29:29,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:29:29,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:29:29,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:29:29,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:29:29,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.71 MB 2025-02-14 22:29:29,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33135.31 MB 2025-02-14 22:29:29,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 22:29:29,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38040.24 MB 2025-02-14 22:29:29,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46428.85 MB 2025-02-14 22:29:29,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 22:29:29,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33135.31 MB 2025-02-14 22:29:29,648 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 22:29:29,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:29:29,649 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:29:29,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:29:29,650 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:29:29,655 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:29:29,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:29:29,656 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:29:29,656 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:30:41,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:30:41,205 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:30:41,210 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:30:41,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:30:41,214 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 963, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:30:41,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:30:41,215 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 963, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:30:56,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:30:56,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:30:56,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.84 seconds 2025-02-14 22:30:56,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:56,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19679.04 MB 2025-02-14 22:30:56,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23087.05 MB 2025-02-14 22:30:56,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3408.00 MB 2025-02-14 22:30:56,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54817.46 MB 2025-02-14 22:30:56,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25654.46 MB 2025-02-14 22:30:56,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29163.00 MB 2025-02-14 22:30:56,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32095.62 MB 2025-02-14 22:30:56,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:30:56,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:30:56,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:30:56,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:56,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23087.05 MB 2025-02-14 22:30:56,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20785.24 MB 2025-02-14 22:30:56,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2301.81 MB 2025-02-14 22:30:56,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25654.46 MB 2025-02-14 22:30:56,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41188.07 MB 2025-02-14 22:30:56,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15533.60 MB 2025-02-14 22:30:56,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34074.23 MB 2025-02-14 22:30:58,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:30:58,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:30:58,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 22:30:58,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20785.24 MB 2025-02-14 22:30:58,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21316.08 MB 2025-02-14 22:30:58,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:30:58,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41188.07 MB 2025-02-14 22:30:58,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24368.91 MB 2025-02-14 22:30:58,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16819.16 MB 2025-02-14 22:30:58,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25295.66 MB 2025-02-14 22:30:58,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:30:58,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:30:58,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:30:58,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21316.08 MB 2025-02-14 22:30:58,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23205.61 MB 2025-02-14 22:30:58,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:30:58,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24368.91 MB 2025-02-14 22:30:58,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26256.34 MB 2025-02-14 22:30:58,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:30:58,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24623.04 MB 2025-02-14 22:30:58,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:30:58,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:30:58,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:30:58,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23205.61 MB 2025-02-14 22:30:58,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25447.47 MB 2025-02-14 22:30:58,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:30:58,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26256.34 MB 2025-02-14 22:30:58,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32862.37 MB 2025-02-14 22:30:58,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:30:58,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30991.75 MB 2025-02-14 22:30:58,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:30:58,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:30:58,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:30:58,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21316.08 MB 2025-02-14 22:30:58,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25447.47 MB 2025-02-14 22:30:58,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:30:58,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24368.91 MB 2025-02-14 22:30:58,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32862.37 MB 2025-02-14 22:30:58,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 22:30:58,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30991.75 MB 2025-02-14 22:30:58,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:30:58,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:30:58,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:30:58,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26981.01 MB 2025-02-14 22:30:58,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27748.01 MB 2025-02-14 22:30:58,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:30:58,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32862.37 MB 2025-02-14 22:30:58,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33277.61 MB 2025-02-14 22:30:58,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:30:58,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28455.80 MB 2025-02-14 22:30:58,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:30:58,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:30:58,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:30:58,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28160.90 MB 2025-02-14 22:30:58,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28389.57 MB 2025-02-14 22:30:58,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 22:30:58,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33277.61 MB 2025-02-14 22:30:58,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33277.61 MB 2025-02-14 22:30:58,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:30:58,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28628.74 MB 2025-02-14 22:30:58,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:30:58,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:30:58,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.25 seconds 2025-02-14 22:30:58,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16323.88 MB 2025-02-14 22:30:58,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28590.15 MB 2025-02-14 22:30:58,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12266.27 MB 2025-02-14 22:30:58,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54817.46 MB 2025-02-14 22:30:58,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33277.61 MB 2025-02-14 22:30:58,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21539.85 MB 2025-02-14 22:30:58,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28628.74 MB 2025-02-14 22:30:58,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:30:58,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:30:58,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:30:58,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28590.15 MB 2025-02-14 22:30:58,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21320.65 MB 2025-02-14 22:30:58,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7269.50 MB 2025-02-14 22:30:58,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33277.61 MB 2025-02-14 22:30:58,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33277.61 MB 2025-02-14 22:30:58,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:30:58,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31095.67 MB 2025-02-14 22:30:58,752 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 22:30:58,752 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:30:58,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:30:58,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:30:58,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:30:58,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:30:58,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21320.65 MB 2025-02-14 22:30:58,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29738.80 MB 2025-02-14 22:30:58,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 22:30:58,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33277.61 MB 2025-02-14 22:30:58,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41647.34 MB 2025-02-14 22:30:58,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 22:30:58,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29738.80 MB 2025-02-14 22:30:58,914 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 22:30:58,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:30:58,915 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:30:58,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:30:58,916 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:30:58,921 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:30:58,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:30:58,922 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:30:58,922 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:32:21,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:21,538 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:32:21,543 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:32:21,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:21,547 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1606, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:32:21,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:21,548 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1606, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:32:46,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:32:46,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:32:46,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.70 seconds 2025-02-14 22:32:46,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:46,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24159.57 MB 2025-02-14 22:32:46,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29843.12 MB 2025-02-14 22:32:46,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5683.54 MB 2025-02-14 22:32:46,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54200.89 MB 2025-02-14 22:32:46,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36297.51 MB 2025-02-14 22:32:46,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17903.39 MB 2025-02-14 22:32:46,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38841.08 MB 2025-02-14 22:32:46,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:32:46,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:32:46,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:32:46,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:46,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29843.12 MB 2025-02-14 22:32:46,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24126.94 MB 2025-02-14 22:32:46,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5716.17 MB 2025-02-14 22:32:46,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36297.51 MB 2025-02-14 22:32:46,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52298.78 MB 2025-02-14 22:32:46,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16001.27 MB 2025-02-14 22:32:46,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45459.03 MB 2025-02-14 22:32:48,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:32:48,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:32:48,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:32:48,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24126.94 MB 2025-02-14 22:32:48,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24657.79 MB 2025-02-14 22:32:48,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:32:48,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52298.78 MB 2025-02-14 22:32:48,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32027.71 MB 2025-02-14 22:32:48,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20271.07 MB 2025-02-14 22:32:48,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28636.33 MB 2025-02-14 22:32:48,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:32:48,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:32:48,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:32:48,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24657.79 MB 2025-02-14 22:32:48,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26547.32 MB 2025-02-14 22:32:48,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:32:48,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32027.71 MB 2025-02-14 22:32:48,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32027.71 MB 2025-02-14 22:32:48,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:32:48,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27964.75 MB 2025-02-14 22:32:48,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:32:48,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:32:48,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:32:48,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26547.32 MB 2025-02-14 22:32:48,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28789.18 MB 2025-02-14 22:32:48,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:32:48,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32027.71 MB 2025-02-14 22:32:48,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36274.44 MB 2025-02-14 22:32:48,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 22:32:48,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34333.46 MB 2025-02-14 22:32:48,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:32:48,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:32:48,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:32:48,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24657.79 MB 2025-02-14 22:32:48,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28789.18 MB 2025-02-14 22:32:48,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:32:48,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32027.71 MB 2025-02-14 22:32:48,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36274.44 MB 2025-02-14 22:32:48,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 22:32:48,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34333.46 MB 2025-02-14 22:32:48,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:32:48,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:32:48,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:32:48,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30322.72 MB 2025-02-14 22:32:48,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31089.72 MB 2025-02-14 22:32:48,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:32:48,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36274.44 MB 2025-02-14 22:32:48,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36689.67 MB 2025-02-14 22:32:48,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:32:48,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31797.51 MB 2025-02-14 22:32:48,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:32:48,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:32:48,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:32:48,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31502.61 MB 2025-02-14 22:32:48,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31730.92 MB 2025-02-14 22:32:48,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.31 MB 2025-02-14 22:32:48,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36689.67 MB 2025-02-14 22:32:48,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36689.67 MB 2025-02-14 22:32:48,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:32:48,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31941.15 MB 2025-02-14 22:32:48,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:32:48,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:32:48,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.14 seconds 2025-02-14 22:32:48,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18564.14 MB 2025-02-14 22:32:48,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31931.75 MB 2025-02-14 22:32:48,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13367.61 MB 2025-02-14 22:32:48,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54200.89 MB 2025-02-14 22:32:48,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36689.67 MB 2025-02-14 22:32:48,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17511.22 MB 2025-02-14 22:32:48,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31941.15 MB 2025-02-14 22:32:48,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:32:48,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:32:48,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:32:48,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31931.75 MB 2025-02-14 22:32:48,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23564.72 MB 2025-02-14 22:32:48,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8367.03 MB 2025-02-14 22:32:48,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36689.67 MB 2025-02-14 22:32:48,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36689.67 MB 2025-02-14 22:32:48,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:32:48,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34440.34 MB 2025-02-14 22:32:48,980 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 22:32:48,980 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:32:48,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:32:48,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:32:48,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:32:48,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:32:48,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23564.72 MB 2025-02-14 22:32:48,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31993.84 MB 2025-02-14 22:32:48,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 22:32:48,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36689.67 MB 2025-02-14 22:32:48,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45069.89 MB 2025-02-14 22:32:48,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 22:32:48,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31993.84 MB 2025-02-14 22:32:49,142 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 22:32:49,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:49,144 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:32:49,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:49,145 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:32:49,149 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:32:49,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:49,150 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:32:49,150 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:32:59,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:59,616 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:32:59,625 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:32:59,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:59,632 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1743, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:32:59,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:32:59,634 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1743, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:33:26,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:33:26,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:33:26,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.30 seconds 2025-02-14 22:33:26,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:26,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25114.21 MB 2025-02-14 22:33:26,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31282.59 MB 2025-02-14 22:33:26,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6168.38 MB 2025-02-14 22:33:26,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53450.11 MB 2025-02-14 22:33:26,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36779.85 MB 2025-02-14 22:33:26,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16670.26 MB 2025-02-14 22:33:26,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40248.70 MB 2025-02-14 22:33:27,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:33:27,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:33:27,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:33:27,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:27,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31282.59 MB 2025-02-14 22:33:27,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24839.16 MB 2025-02-14 22:33:27,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6443.42 MB 2025-02-14 22:33:27,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36779.85 MB 2025-02-14 22:33:27,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57912.85 MB 2025-02-14 22:33:27,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21133.00 MB 2025-02-14 22:33:27,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48880.72 MB 2025-02-14 22:33:29,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:33:29,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:33:29,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 22:33:29,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24839.16 MB 2025-02-14 22:33:29,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25370.01 MB 2025-02-14 22:33:29,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:33:29,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57912.85 MB 2025-02-14 22:33:29,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27841.79 MB 2025-02-14 22:33:29,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30071.06 MB 2025-02-14 22:33:29,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29349.59 MB 2025-02-14 22:33:29,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:33:29,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:33:29,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:33:29,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25370.01 MB 2025-02-14 22:33:29,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27259.54 MB 2025-02-14 22:33:29,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:33:29,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27841.79 MB 2025-02-14 22:33:29,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30672.95 MB 2025-02-14 22:33:29,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 22:33:29,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28676.97 MB 2025-02-14 22:33:29,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:33:29,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:33:29,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:33:29,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27259.54 MB 2025-02-14 22:33:29,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.40 MB 2025-02-14 22:33:29,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:33:29,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30672.95 MB 2025-02-14 22:33:29,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36807.11 MB 2025-02-14 22:33:29,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:33:29,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35045.68 MB 2025-02-14 22:33:29,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:33:29,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:33:29,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:33:29,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25370.01 MB 2025-02-14 22:33:29,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.40 MB 2025-02-14 22:33:29,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:33:29,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27841.79 MB 2025-02-14 22:33:29,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36807.11 MB 2025-02-14 22:33:29,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 22:33:29,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35045.68 MB 2025-02-14 22:33:29,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:33:29,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:33:29,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:33:29,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31034.94 MB 2025-02-14 22:33:29,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31801.94 MB 2025-02-14 22:33:29,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:33:29,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36807.11 MB 2025-02-14 22:33:29,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37224.45 MB 2025-02-14 22:33:29,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:33:29,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32509.73 MB 2025-02-14 22:33:29,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:33:29,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:33:29,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:33:29,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32214.83 MB 2025-02-14 22:33:29,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32443.09 MB 2025-02-14 22:33:29,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.26 MB 2025-02-14 22:33:29,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37224.45 MB 2025-02-14 22:33:29,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37224.45 MB 2025-02-14 22:33:29,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:29,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32670.64 MB 2025-02-14 22:33:29,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:33:29,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:33:29,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.80 seconds 2025-02-14 22:33:29,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19041.46 MB 2025-02-14 22:33:29,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32643.35 MB 2025-02-14 22:33:29,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13601.89 MB 2025-02-14 22:33:29,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53450.11 MB 2025-02-14 22:33:29,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37224.45 MB 2025-02-14 22:33:29,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16225.67 MB 2025-02-14 22:33:29,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32670.64 MB 2025-02-14 22:33:29,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:33:29,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:33:29,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:33:29,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32643.35 MB 2025-02-14 22:33:29,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24033.50 MB 2025-02-14 22:33:29,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8609.85 MB 2025-02-14 22:33:29,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37224.45 MB 2025-02-14 22:33:29,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37224.45 MB 2025-02-14 22:33:29,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:29,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35145.10 MB 2025-02-14 22:33:29,726 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 22:33:29,727 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:33:29,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:33:29,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:33:29,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:33:29,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:29,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24033.50 MB 2025-02-14 22:33:29,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32438.61 MB 2025-02-14 22:33:29,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 22:33:29,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37224.45 MB 2025-02-14 22:33:29,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45581.60 MB 2025-02-14 22:33:29,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 22:33:29,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32438.61 MB 2025-02-14 22:33:29,890 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 22:33:29,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:29,891 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:33:29,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:29,892 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:33:29,897 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:33:29,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:29,898 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:33:29,898 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:33:39,316 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:39,316 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:33:39,321 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:33:39,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:39,324 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 160, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:33:39,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:39,325 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 160, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:33:41,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:33:41,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:33:41,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.52 seconds 2025-02-14 22:33:41,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:41,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14083.61 MB 2025-02-14 22:33:41,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14649.84 MB 2025-02-14 22:33:41,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 566.23 MB 2025-02-14 22:33:41,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58116.28 MB 2025-02-14 22:33:41,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 22:33:41,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38587.60 MB 2025-02-14 22:33:41,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23554.98 MB 2025-02-14 22:33:41,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:33:41,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:33:41,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:33:41,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:41,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14649.84 MB 2025-02-14 22:33:41,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14924.18 MB 2025-02-14 22:33:41,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 274.34 MB 2025-02-14 22:33:41,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 22:33:41,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 22:33:41,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:41,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16953.90 MB 2025-02-14 22:33:42,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:33:42,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:33:42,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 22:33:42,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:42,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14924.18 MB 2025-02-14 22:33:42,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15136.52 MB 2025-02-14 22:33:42,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.34 MB 2025-02-14 22:33:42,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 22:33:42,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 22:33:42,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:42,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19094.87 MB 2025-02-14 22:33:42,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:33:42,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:33:42,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:33:42,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:42,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15136.45 MB 2025-02-14 22:33:42,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15892.08 MB 2025-02-14 22:33:42,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 755.63 MB 2025-02-14 22:33:42,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 22:33:42,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 22:33:42,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:42,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16459.06 MB 2025-02-14 22:33:42,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:33:42,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:33:42,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:33:42,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:42,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15892.08 MB 2025-02-14 22:33:42,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16788.86 MB 2025-02-14 22:33:42,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 896.78 MB 2025-02-14 22:33:42,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 22:33:42,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20283.65 MB 2025-02-14 22:33:42,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 754.97 MB 2025-02-14 22:33:42,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19006.54 MB 2025-02-14 22:33:42,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:33:42,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:33:42,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:33:42,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:42,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15136.45 MB 2025-02-14 22:33:42,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16788.86 MB 2025-02-14 22:33:42,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1652.41 MB 2025-02-14 22:33:42,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 22:33:42,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20283.65 MB 2025-02-14 22:33:42,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 754.97 MB 2025-02-14 22:33:42,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19006.54 MB 2025-02-14 22:33:42,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:33:42,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:33:42,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 22:33:42,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:42,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17402.28 MB 2025-02-14 22:33:42,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17709.08 MB 2025-02-14 22:33:42,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.80 MB 2025-02-14 22:33:42,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20283.65 MB 2025-02-14 22:33:42,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 22:33:42,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 22:33:42,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18000.38 MB 2025-02-14 22:33:42,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:33:42,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:33:42,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:33:42,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:42,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17874.24 MB 2025-02-14 22:33:42,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18102.31 MB 2025-02-14 22:33:42,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.07 MB 2025-02-14 22:33:42,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20449.33 MB 2025-02-14 22:33:42,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 22:33:42,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:42,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18125.31 MB 2025-02-14 22:33:42,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:33:42,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:33:42,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.49 seconds 2025-02-14 22:33:42,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:42,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13526.16 MB 2025-02-14 22:33:42,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18302.94 MB 2025-02-14 22:33:42,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4776.78 MB 2025-02-14 22:33:42,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58116.28 MB 2025-02-14 22:33:42,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 22:33:42,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37666.95 MB 2025-02-14 22:33:42,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18302.94 MB 2025-02-14 22:33:43,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:33:43,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:33:43,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:33:43,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:43,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18302.94 MB 2025-02-14 22:33:43,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17391.06 MB 2025-02-14 22:33:43,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -911.88 MB 2025-02-14 22:33:43,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20449.33 MB 2025-02-14 22:33:43,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 22:33:43,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:43,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19104.91 MB 2025-02-14 22:33:43,101 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 22:33:43,102 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:33:43,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:33:43,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:33:43,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:33:43,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:43,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17391.06 MB 2025-02-14 22:33:43,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25811.84 MB 2025-02-14 22:33:43,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 22:33:43,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20449.33 MB 2025-02-14 22:33:43,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28821.16 MB 2025-02-14 22:33:43,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 22:33:43,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25811.84 MB 2025-02-14 22:33:43,263 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 22:33:43,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:43,264 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:33:43,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:43,265 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:33:43,270 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:33:43,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:43,271 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:33:43,271 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 22:33:52,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:52,875 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:33:52,879 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:33:52,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:52,883 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:33:52,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:52,884 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:33:55,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:33:55,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:33:55,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.39 seconds 2025-02-14 22:33:55,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:55,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-14 22:33:55,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-14 22:33:55,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 22:33:55,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37192.99 MB 2025-02-14 22:33:55,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19339.94 MB 2025-02-14 22:33:55,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17853.05 MB 2025-02-14 22:33:55,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23506.21 MB 2025-02-14 22:33:55,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:33:55,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:33:55,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:33:55,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:55,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-14 22:33:55,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14712.21 MB 2025-02-14 22:33:55,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 135.92 MB 2025-02-14 22:33:55,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19339.94 MB 2025-02-14 22:33:55,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19339.94 MB 2025-02-14 22:33:55,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:55,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16504.43 MB 2025-02-14 22:33:55,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:33:55,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:33:55,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.66 seconds 2025-02-14 22:33:55,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:55,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14712.21 MB 2025-02-14 22:33:55,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14891.37 MB 2025-02-14 22:33:55,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 179.16 MB 2025-02-14 22:33:55,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19339.94 MB 2025-02-14 22:33:55,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18962.45 MB 2025-02-14 22:33:55,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -377.49 MB 2025-02-14 22:33:55,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18882.90 MB 2025-02-14 22:33:55,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:33:55,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:33:55,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:33:55,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:55,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14891.31 MB 2025-02-14 22:33:55,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15528.87 MB 2025-02-14 22:33:55,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.56 MB 2025-02-14 22:33:55,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18962.45 MB 2025-02-14 22:33:55,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18962.45 MB 2025-02-14 22:33:55,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:55,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16007.26 MB 2025-02-14 22:33:56,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:33:56,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:33:56,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:33:56,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:56,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15528.87 MB 2025-02-14 22:33:56,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16285.54 MB 2025-02-14 22:33:56,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 756.67 MB 2025-02-14 22:33:56,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18962.45 MB 2025-02-14 22:33:56,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19281.22 MB 2025-02-14 22:33:56,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 318.77 MB 2025-02-14 22:33:56,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18156.95 MB 2025-02-14 22:33:56,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:33:56,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:33:56,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:33:56,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:56,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14891.31 MB 2025-02-14 22:33:56,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16285.54 MB 2025-02-14 22:33:56,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1394.23 MB 2025-02-14 22:33:56,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18962.45 MB 2025-02-14 22:33:56,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19281.22 MB 2025-02-14 22:33:56,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 318.77 MB 2025-02-14 22:33:56,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18156.95 MB 2025-02-14 22:33:56,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:33:56,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:33:56,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 22:33:56,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:56,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16803.11 MB 2025-02-14 22:33:56,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17061.97 MB 2025-02-14 22:33:56,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 258.86 MB 2025-02-14 22:33:56,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19281.22 MB 2025-02-14 22:33:56,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19419.63 MB 2025-02-14 22:33:56,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 138.41 MB 2025-02-14 22:33:56,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17312.14 MB 2025-02-14 22:33:56,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:33:56,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:33:56,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:33:56,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:56,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17201.33 MB 2025-02-14 22:33:56,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17423.42 MB 2025-02-14 22:33:56,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.09 MB 2025-02-14 22:33:56,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19419.63 MB 2025-02-14 22:33:56,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19419.63 MB 2025-02-14 22:33:56,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:56,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17423.42 MB 2025-02-14 22:33:56,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:33:56,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:33:56,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.21 seconds 2025-02-14 22:33:56,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:56,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-14 22:33:56,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14240.87 MB 2025-02-14 22:33:56,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 739.10 MB 2025-02-14 22:33:56,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37192.99 MB 2025-02-14 22:33:56,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19419.63 MB 2025-02-14 22:33:56,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17773.36 MB 2025-02-14 22:33:56,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17623.22 MB 2025-02-14 22:33:56,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:33:56,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:33:56,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:33:56,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:56,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14240.87 MB 2025-02-14 22:33:56,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17236.84 MB 2025-02-14 22:33:56,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2995.97 MB 2025-02-14 22:33:56,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19419.63 MB 2025-02-14 22:33:56,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19419.63 MB 2025-02-14 22:33:56,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:33:56,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17536.29 MB 2025-02-14 22:33:56,380 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8110, cut from 8112 2025-02-14 22:33:56,380 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 22:33:56,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:33:56,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:33:56,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:33:56,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:33:56,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17236.84 MB 2025-02-14 22:33:56,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25621.86 MB 2025-02-14 22:33:56,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8385.02 MB 2025-02-14 22:33:56,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19419.63 MB 2025-02-14 22:33:56,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29842.47 MB 2025-02-14 22:33:56,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10422.85 MB 2025-02-14 22:33:56,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25621.86 MB 2025-02-14 22:33:56,541 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7902] 2025-02-14 22:33:56,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:56,543 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:33:56,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:56,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:33:56,548 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:33:56,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:33:56,550 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:33:56,550 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 22:34:06,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:34:06,284 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:34:06,289 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:34:06,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:34:06,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:34:06,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:34:06,293 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:34:08,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:34:08,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:34:08,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.50 seconds 2025-02-14 22:34:08,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:08,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19475.65 MB 2025-02-14 22:34:08,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20045.42 MB 2025-02-14 22:34:08,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 22:34:08,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38180.75 MB 2025-02-14 22:34:08,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23450.35 MB 2025-02-14 22:34:08,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14730.40 MB 2025-02-14 22:34:08,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28947.83 MB 2025-02-14 22:34:08,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:34:08,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:34:08,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:34:08,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:08,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20045.42 MB 2025-02-14 22:34:08,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20322.06 MB 2025-02-14 22:34:08,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.64 MB 2025-02-14 22:34:08,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23450.35 MB 2025-02-14 22:34:08,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24020.78 MB 2025-02-14 22:34:08,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 570.43 MB 2025-02-14 22:34:08,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22311.03 MB 2025-02-14 22:34:09,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:34:09,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:34:09,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 22:34:09,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:09,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20322.06 MB 2025-02-14 22:34:09,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20535.73 MB 2025-02-14 22:34:09,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 22:34:09,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24020.78 MB 2025-02-14 22:34:09,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24020.78 MB 2025-02-14 22:34:09,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:34:09,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24492.75 MB 2025-02-14 22:34:09,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:34:09,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:34:09,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:34:09,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:09,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20535.73 MB 2025-02-14 22:34:09,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21296.08 MB 2025-02-14 22:34:09,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 22:34:09,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24020.78 MB 2025-02-14 22:34:09,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24020.78 MB 2025-02-14 22:34:09,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:34:09,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21866.60 MB 2025-02-14 22:34:09,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:34:09,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:34:09,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:34:09,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:09,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21296.08 MB 2025-02-14 22:34:09,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22198.47 MB 2025-02-14 22:34:09,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 22:34:09,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24020.78 MB 2025-02-14 22:34:09,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25738.35 MB 2025-02-14 22:34:09,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1717.57 MB 2025-02-14 22:34:09,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24430.92 MB 2025-02-14 22:34:09,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:34:09,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:34:09,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:34:09,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:09,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20535.73 MB 2025-02-14 22:34:09,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22198.47 MB 2025-02-14 22:34:09,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 22:34:09,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24020.78 MB 2025-02-14 22:34:09,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25738.35 MB 2025-02-14 22:34:09,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1717.57 MB 2025-02-14 22:34:09,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24430.92 MB 2025-02-14 22:34:09,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:34:09,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:34:09,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 22:34:09,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:09,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22815.72 MB 2025-02-14 22:34:09,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23125.35 MB 2025-02-14 22:34:09,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-14 22:34:09,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25738.35 MB 2025-02-14 22:34:09,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25906.12 MB 2025-02-14 22:34:09,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 22:34:09,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23418.47 MB 2025-02-14 22:34:09,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:34:09,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:34:09,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:34:09,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:09,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23291.55 MB 2025-02-14 22:34:09,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23519.84 MB 2025-02-14 22:34:09,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.29 MB 2025-02-14 22:34:09,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25906.12 MB 2025-02-14 22:34:09,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25906.12 MB 2025-02-14 22:34:09,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:34:09,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23538.80 MB 2025-02-14 22:34:09,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:34:09,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:34:09,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.47 seconds 2025-02-14 22:34:09,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:09,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18914.71 MB 2025-02-14 22:34:09,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23720.91 MB 2025-02-14 22:34:09,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4806.20 MB 2025-02-14 22:34:09,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38180.75 MB 2025-02-14 22:34:09,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25906.12 MB 2025-02-14 22:34:09,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12274.63 MB 2025-02-14 22:34:09,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23720.91 MB 2025-02-14 22:34:10,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:34:10,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:34:10,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:34:10,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:10,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23720.91 MB 2025-02-14 22:34:10,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22792.11 MB 2025-02-14 22:34:10,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -928.80 MB 2025-02-14 22:34:10,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25906.12 MB 2025-02-14 22:34:10,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25906.12 MB 2025-02-14 22:34:10,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:34:10,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24524.65 MB 2025-02-14 22:34:10,048 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:34:10,049 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 22:34:10,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:34:10,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:34:10,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:34:10,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:34:10,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22792.11 MB 2025-02-14 22:34:10,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31231.13 MB 2025-02-14 22:34:10,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:34:10,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25906.12 MB 2025-02-14 22:34:10,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36396.07 MB 2025-02-14 22:34:10,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 22:34:10,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31231.13 MB 2025-02-14 22:34:10,210 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:34:10,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:34:10,212 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:34:10,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:34:10,213 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:34:10,217 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:34:10,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:34:10,218 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:34:10,218 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 22:35:35,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:35:35,391 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:35:35,399 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:35:35,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:35:35,407 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:35:35,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:35:35,409 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:35:38,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:35:38,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:35:38,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.81 seconds 2025-02-14 22:35:38,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:38,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-14 22:35:38,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-14 22:35:38,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-14 22:35:38,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48981.08 MB 2025-02-14 22:35:38,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22284.34 MB 2025-02-14 22:35:38,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26696.74 MB 2025-02-14 22:35:38,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23701.31 MB 2025-02-14 22:35:38,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:35:38,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:35:38,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:35:38,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:38,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-14 22:35:38,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15089.54 MB 2025-02-14 22:35:38,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.04 MB 2025-02-14 22:35:38,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22284.34 MB 2025-02-14 22:35:38,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22284.34 MB 2025-02-14 22:35:38,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:35:38,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17248.77 MB 2025-02-14 22:35:39,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:35:39,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:35:39,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.80 seconds 2025-02-14 22:35:39,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15089.54 MB 2025-02-14 22:35:39,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15312.49 MB 2025-02-14 22:35:39,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.95 MB 2025-02-14 22:35:39,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22284.34 MB 2025-02-14 22:35:39,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21906.85 MB 2025-02-14 22:35:39,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -377.49 MB 2025-02-14 22:35:39,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19259.19 MB 2025-02-14 22:35:39,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:35:39,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:35:39,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:35:39,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.42 MB 2025-02-14 22:35:39,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16105.84 MB 2025-02-14 22:35:39,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 793.41 MB 2025-02-14 22:35:39,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21906.85 MB 2025-02-14 22:35:39,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21906.85 MB 2025-02-14 22:35:39,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:35:39,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16701.16 MB 2025-02-14 22:35:39,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:35:39,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:35:39,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:35:39,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16105.84 MB 2025-02-14 22:35:39,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17047.45 MB 2025-02-14 22:35:39,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 941.62 MB 2025-02-14 22:35:39,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21906.85 MB 2025-02-14 22:35:39,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21906.85 MB 2025-02-14 22:35:39,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:35:39,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19376.01 MB 2025-02-14 22:35:39,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:35:39,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:35:39,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 22:35:39,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.42 MB 2025-02-14 22:35:39,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17047.45 MB 2025-02-14 22:35:39,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1735.03 MB 2025-02-14 22:35:39,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21906.85 MB 2025-02-14 22:35:39,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21906.85 MB 2025-02-14 22:35:39,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:35:39,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19376.01 MB 2025-02-14 22:35:39,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:35:39,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:35:39,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:35:39,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17691.54 MB 2025-02-14 22:35:39,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18013.68 MB 2025-02-14 22:35:39,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.14 MB 2025-02-14 22:35:39,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21906.85 MB 2025-02-14 22:35:39,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22080.91 MB 2025-02-14 22:35:39,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-14 22:35:39,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18318.09 MB 2025-02-14 22:35:39,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:35:39,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:35:39,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:35:39,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18187.10 MB 2025-02-14 22:35:39,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18413.99 MB 2025-02-14 22:35:39,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.89 MB 2025-02-14 22:35:39,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22080.91 MB 2025-02-14 22:35:39,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22080.91 MB 2025-02-14 22:35:39,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:35:39,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18439.38 MB 2025-02-14 22:35:39,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:35:39,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:35:39,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.82 seconds 2025-02-14 22:35:39,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-14 22:35:39,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18614.58 MB 2025-02-14 22:35:39,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5015.25 MB 2025-02-14 22:35:39,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48981.08 MB 2025-02-14 22:35:39,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22080.91 MB 2025-02-14 22:35:39,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26900.17 MB 2025-02-14 22:35:39,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18614.58 MB 2025-02-14 22:35:39,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:35:39,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:35:39,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:35:39,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18614.58 MB 2025-02-14 22:35:39,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17501.22 MB 2025-02-14 22:35:39,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1113.36 MB 2025-02-14 22:35:39,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22080.91 MB 2025-02-14 22:35:39,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22080.91 MB 2025-02-14 22:35:39,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:35:39,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19215.90 MB 2025-02-14 22:35:39,515 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 22:35:39,516 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 22:35:39,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:35:39,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:35:39,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:35:39,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:35:39,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17501.22 MB 2025-02-14 22:35:39,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25919.37 MB 2025-02-14 22:35:39,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 22:35:39,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22080.91 MB 2025-02-14 22:35:39,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30450.65 MB 2025-02-14 22:35:39,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 22:35:39,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25919.37 MB 2025-02-14 22:35:39,680 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 22:35:39,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:35:39,682 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:35:39,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:35:39,683 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:35:39,687 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:35:39,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:35:39,689 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:35:39,689 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 22:36:41,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:36:41,151 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:36:41,156 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:36:41,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:36:41,159 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1960, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:36:41,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:36:41,160 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1960, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:37:11,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:37:11,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:37:11,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.11 seconds 2025-02-14 22:37:11,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:11,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26626.30 MB 2025-02-14 22:37:11,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33563.68 MB 2025-02-14 22:37:11,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6937.38 MB 2025-02-14 22:37:11,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43004.20 MB 2025-02-14 22:37:11,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37601.94 MB 2025-02-14 22:37:11,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5402.26 MB 2025-02-14 22:37:11,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42440.27 MB 2025-02-14 22:37:11,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:37:11,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:37:11,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 22:37:11,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:11,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33563.68 MB 2025-02-14 22:37:11,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25967.28 MB 2025-02-14 22:37:11,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7596.40 MB 2025-02-14 22:37:11,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37601.94 MB 2025-02-14 22:37:11,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62696.46 MB 2025-02-14 22:37:11,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25094.52 MB 2025-02-14 22:37:11,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53051.23 MB 2025-02-14 22:37:13,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:37:13,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:37:13,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 22:37:13,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:13,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25967.28 MB 2025-02-14 22:37:13,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26498.12 MB 2025-02-14 22:37:13,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:37:13,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62696.46 MB 2025-02-14 22:37:13,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 22:37:13,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30616.32 MB 2025-02-14 22:37:13,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30476.67 MB 2025-02-14 22:37:13,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:37:13,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:37:13,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:37:13,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:13,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 22:37:13,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28387.65 MB 2025-02-14 22:37:13,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:37:13,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 22:37:13,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32080.13 MB 2025-02-14 22:37:13,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:37:13,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29805.08 MB 2025-02-14 22:37:13,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:37:13,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:37:13,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:37:13,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:13,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28387.65 MB 2025-02-14 22:37:13,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 22:37:13,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:37:13,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 22:37:13,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37742.44 MB 2025-02-14 22:37:13,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:37:13,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 22:37:13,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:37:13,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:37:13,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:37:13,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:13,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 22:37:13,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 22:37:13,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:37:13,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32080.13 MB 2025-02-14 22:37:13,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37742.44 MB 2025-02-14 22:37:13,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 22:37:13,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 22:37:13,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:37:13,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:37:13,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:37:13,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:13,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32163.05 MB 2025-02-14 22:37:13,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32930.05 MB 2025-02-14 22:37:13,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:37:13,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37742.44 MB 2025-02-14 22:37:13,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 22:37:13,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:37:13,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33637.84 MB 2025-02-14 22:37:13,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:37:13,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:37:13,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:37:13,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:13,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33342.94 MB 2025-02-14 22:37:13,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33572.00 MB 2025-02-14 22:37:13,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-14 22:37:13,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38159.78 MB 2025-02-14 22:37:13,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 22:37:13,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:37:13,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33755.68 MB 2025-02-14 22:37:13,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:37:13,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:37:13,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.58 seconds 2025-02-14 22:37:13,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:13,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19797.50 MB 2025-02-14 22:37:13,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33772.98 MB 2025-02-14 22:37:13,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13975.47 MB 2025-02-14 22:37:13,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43004.20 MB 2025-02-14 22:37:13,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 22:37:13,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4844.42 MB 2025-02-14 22:37:13,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.98 MB 2025-02-14 22:37:14,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:37:14,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:37:14,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:37:14,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:14,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33772.98 MB 2025-02-14 22:37:14,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24800.37 MB 2025-02-14 22:37:14,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8972.61 MB 2025-02-14 22:37:14,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38159.78 MB 2025-02-14 22:37:14,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 22:37:14,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:37:14,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36283.42 MB 2025-02-14 22:37:14,032 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 22:37:14,032 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:37:14,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:37:14,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:37:14,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:37:14,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:37:14,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24800.37 MB 2025-02-14 22:37:14,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33235.22 MB 2025-02-14 22:37:14,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 22:37:14,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38159.78 MB 2025-02-14 22:37:14,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46546.29 MB 2025-02-14 22:37:14,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 22:37:14,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33235.22 MB 2025-02-14 22:37:14,195 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 22:37:14,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:37:14,196 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:37:14,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:37:14,197 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:37:14,202 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:37:14,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:37:14,203 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:37:14,203 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:38:11,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:38:11,534 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:38:11,542 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:38:11,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:38:11,549 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1285, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:38:11,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:38:11,551 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1285, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:38:31,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:38:31,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:38:31,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.80 seconds 2025-02-14 22:38:31,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:31,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21922.79 MB 2025-02-14 22:38:31,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.34 MB 2025-02-14 22:38:31,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4547.54 MB 2025-02-14 22:38:31,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59125.01 MB 2025-02-14 22:38:31,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35215.38 MB 2025-02-14 22:38:31,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23909.63 MB 2025-02-14 22:38:31,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.03 MB 2025-02-14 22:38:31,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:38:31,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:38:31,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:38:31,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:31,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26470.34 MB 2025-02-14 22:38:31,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.17 MB 2025-02-14 22:38:31,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4012.17 MB 2025-02-14 22:38:31,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35215.38 MB 2025-02-14 22:38:31,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44029.71 MB 2025-02-14 22:38:31,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8814.33 MB 2025-02-14 22:38:31,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39565.49 MB 2025-02-14 22:38:33,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:38:33,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:38:33,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:38:33,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:33,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.17 MB 2025-02-14 22:38:33,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22989.01 MB 2025-02-14 22:38:33,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:38:33,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44029.71 MB 2025-02-14 22:38:33,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 22:38:33,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17546.87 MB 2025-02-14 22:38:33,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26968.59 MB 2025-02-14 22:38:33,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:38:33,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:38:33,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:38:33,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:33,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 22:38:33,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.54 MB 2025-02-14 22:38:33,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:38:33,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 22:38:33,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 22:38:33,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:38:33,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26295.97 MB 2025-02-14 22:38:33,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:38:33,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:38:33,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:38:33,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:33,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.54 MB 2025-02-14 22:38:33,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 22:38:33,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:38:33,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27426.55 MB 2025-02-14 22:38:33,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34504.44 MB 2025-02-14 22:38:33,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 22:38:33,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 22:38:33,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:38:33,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:38:33,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:38:33,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:33,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 22:38:33,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 22:38:33,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:38:33,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 22:38:33,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34504.44 MB 2025-02-14 22:38:33,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 22:38:33,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 22:38:33,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:38:33,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:38:33,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:38:33,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:33,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28653.94 MB 2025-02-14 22:38:33,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29420.94 MB 2025-02-14 22:38:33,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:38:33,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34504.44 MB 2025-02-14 22:38:33,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34919.68 MB 2025-02-14 22:38:33,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:38:33,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30128.73 MB 2025-02-14 22:38:33,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:38:33,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:38:33,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:38:33,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:33,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29833.83 MB 2025-02-14 22:38:33,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30062.40 MB 2025-02-14 22:38:33,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 22:38:33,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34919.68 MB 2025-02-14 22:38:33,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34919.68 MB 2025-02-14 22:38:33,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:38:33,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30282.78 MB 2025-02-14 22:38:33,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:38:33,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:38:33,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.20 seconds 2025-02-14 22:38:33,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:33,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17445.75 MB 2025-02-14 22:38:33,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30262.88 MB 2025-02-14 22:38:33,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12817.13 MB 2025-02-14 22:38:33,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59125.01 MB 2025-02-14 22:38:33,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34919.68 MB 2025-02-14 22:38:33,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24205.33 MB 2025-02-14 22:38:33,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30282.78 MB 2025-02-14 22:38:34,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:38:34,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:38:34,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:38:34,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:34,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30262.88 MB 2025-02-14 22:38:34,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22441.00 MB 2025-02-14 22:38:34,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7821.88 MB 2025-02-14 22:38:34,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34919.68 MB 2025-02-14 22:38:34,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34919.68 MB 2025-02-14 22:38:34,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:38:34,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32767.17 MB 2025-02-14 22:38:34,043 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 22:38:34,043 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:38:34,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:38:34,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:38:34,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:38:34,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:38:34,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22441.00 MB 2025-02-14 22:38:34,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30854.91 MB 2025-02-14 22:38:34,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.92 MB 2025-02-14 22:38:34,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34919.68 MB 2025-02-14 22:38:34,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39103.50 MB 2025-02-14 22:38:34,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 22:38:34,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30854.91 MB 2025-02-14 22:38:34,208 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 22:38:34,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:38:34,209 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:38:34,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:38:34,210 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:38:34,215 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:38:34,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:38:34,216 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:38:34,216 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:39:27,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:39:27,907 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:39:27,913 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:39:27,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:39:27,918 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1188, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:39:27,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:39:27,919 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1188, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:39:46,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:39:46,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:39:46,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.33 seconds 2025-02-14 22:39:46,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:46,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21246.88 MB 2025-02-14 22:39:46,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25451.67 MB 2025-02-14 22:39:46,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4204.79 MB 2025-02-14 22:39:46,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47466.94 MB 2025-02-14 22:39:46,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30683.43 MB 2025-02-14 22:39:46,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16783.51 MB 2025-02-14 22:39:46,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34342.13 MB 2025-02-14 22:39:46,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:39:46,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:39:46,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:39:46,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:46,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25451.67 MB 2025-02-14 22:39:46,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21953.89 MB 2025-02-14 22:39:46,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3497.78 MB 2025-02-14 22:39:46,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30683.43 MB 2025-02-14 22:39:46,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43891.29 MB 2025-02-14 22:39:46,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13207.86 MB 2025-02-14 22:39:46,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38021.45 MB 2025-02-14 22:39:48,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:39:48,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:39:48,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:39:48,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21953.89 MB 2025-02-14 22:39:48,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22484.73 MB 2025-02-14 22:39:48,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:39:48,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43891.29 MB 2025-02-14 22:39:48,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27894.22 MB 2025-02-14 22:39:48,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15997.08 MB 2025-02-14 22:39:48,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26463.28 MB 2025-02-14 22:39:48,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:39:48,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:39:48,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:39:48,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.73 MB 2025-02-14 22:39:48,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24374.27 MB 2025-02-14 22:39:48,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:39:48,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27894.22 MB 2025-02-14 22:39:48,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27894.22 MB 2025-02-14 22:39:48,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:39:48,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25791.70 MB 2025-02-14 22:39:48,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:39:48,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:39:48,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:39:48,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24374.27 MB 2025-02-14 22:39:48,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26616.12 MB 2025-02-14 22:39:48,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:39:48,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27894.22 MB 2025-02-14 22:39:48,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34028.39 MB 2025-02-14 22:39:48,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:39:48,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32160.41 MB 2025-02-14 22:39:48,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:39:48,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:39:48,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:39:48,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.73 MB 2025-02-14 22:39:48,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26616.12 MB 2025-02-14 22:39:48,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:39:48,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27894.22 MB 2025-02-14 22:39:48,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34028.39 MB 2025-02-14 22:39:48,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:39:48,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32160.41 MB 2025-02-14 22:39:48,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:39:48,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:39:48,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:39:48,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28149.67 MB 2025-02-14 22:39:48,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28916.67 MB 2025-02-14 22:39:48,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:39:48,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34028.39 MB 2025-02-14 22:39:48,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:39:48,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:39:48,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29624.46 MB 2025-02-14 22:39:48,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:39:48,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:39:48,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:39:48,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29329.56 MB 2025-02-14 22:39:48,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.76 MB 2025-02-14 22:39:48,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.21 MB 2025-02-14 22:39:48,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 22:39:48,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:39:48,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:39:48,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29798.85 MB 2025-02-14 22:39:48,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:39:48,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:39:48,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.75 seconds 2025-02-14 22:39:48,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17107.79 MB 2025-02-14 22:39:48,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29759.25 MB 2025-02-14 22:39:48,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12651.45 MB 2025-02-14 22:39:48,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47466.94 MB 2025-02-14 22:39:48,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:39:48,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13023.31 MB 2025-02-14 22:39:48,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29798.85 MB 2025-02-14 22:39:48,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:39:48,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:39:48,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:39:48,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29759.25 MB 2025-02-14 22:39:48,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22099.83 MB 2025-02-14 22:39:48,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7659.41 MB 2025-02-14 22:39:48,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 22:39:48,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34443.62 MB 2025-02-14 22:39:48,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:39:48,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32260.78 MB 2025-02-14 22:39:48,954 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 22:39:48,954 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:39:48,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:39:48,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:39:48,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:39:48,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:39:48,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22099.83 MB 2025-02-14 22:39:48,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30504.95 MB 2025-02-14 22:39:48,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 22:39:48,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34443.62 MB 2025-02-14 22:39:48,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42800.78 MB 2025-02-14 22:39:48,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 22:39:48,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30504.95 MB 2025-02-14 22:39:49,117 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 22:39:49,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:39:49,118 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:39:49,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:39:49,119 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:39:49,124 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:39:49,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:39:49,125 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:39:49,125 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:41:52,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:41:52,962 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:41:52,967 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:41:52,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:41:52,971 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1340, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:41:52,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:41:52,972 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1340, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:42:13,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:42:13,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:42:13,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.45 seconds 2025-02-14 22:42:13,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:13,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22306.04 MB 2025-02-14 22:42:13,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27048.23 MB 2025-02-14 22:42:13,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.18 MB 2025-02-14 22:42:13,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55335.45 MB 2025-02-14 22:42:13,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35393.63 MB 2025-02-14 22:42:13,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19941.82 MB 2025-02-14 22:42:13,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35854.28 MB 2025-02-14 22:42:13,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:42:13,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:42:13,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:42:13,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:13,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.23 MB 2025-02-14 22:42:13,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.09 MB 2025-02-14 22:42:13,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4304.13 MB 2025-02-14 22:42:13,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35393.63 MB 2025-02-14 22:42:13,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42102.42 MB 2025-02-14 22:42:13,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6708.79 MB 2025-02-14 22:42:13,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38024.27 MB 2025-02-14 22:42:15,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:42:15,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:42:15,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:42:15,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:15,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.09 MB 2025-02-14 22:42:15,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23274.93 MB 2025-02-14 22:42:15,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:42:15,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42102.42 MB 2025-02-14 22:42:15,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26472.35 MB 2025-02-14 22:42:15,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15630.07 MB 2025-02-14 22:42:15,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27253.48 MB 2025-02-14 22:42:15,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:42:15,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:42:15,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:42:15,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:15,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 22:42:15,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25164.47 MB 2025-02-14 22:42:15,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:42:15,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26472.35 MB 2025-02-14 22:42:15,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28359.79 MB 2025-02-14 22:42:15,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:42:15,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26581.90 MB 2025-02-14 22:42:15,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:42:15,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:42:15,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:42:15,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:15,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25164.47 MB 2025-02-14 22:42:15,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-14 22:42:15,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:42:15,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28359.79 MB 2025-02-14 22:42:15,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34965.82 MB 2025-02-14 22:42:15,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:42:15,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-14 22:42:15,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:42:15,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:42:15,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 22:42:15,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:15,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 22:42:15,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-14 22:42:15,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:42:15,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26472.35 MB 2025-02-14 22:42:15,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34965.82 MB 2025-02-14 22:42:15,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 22:42:15,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-14 22:42:15,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:42:15,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:42:15,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 22:42:15,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:15,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.87 MB 2025-02-14 22:42:15,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29706.87 MB 2025-02-14 22:42:15,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:42:15,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34965.82 MB 2025-02-14 22:42:15,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35383.15 MB 2025-02-14 22:42:15,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 22:42:15,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30414.66 MB 2025-02-14 22:42:15,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:42:15,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:42:15,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:42:15,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:15,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30119.76 MB 2025-02-14 22:42:15,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30346.16 MB 2025-02-14 22:42:15,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.40 MB 2025-02-14 22:42:15,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35383.15 MB 2025-02-14 22:42:15,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35383.15 MB 2025-02-14 22:42:15,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:42:15,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30579.59 MB 2025-02-14 22:42:15,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:42:15,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:42:15,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.87 seconds 2025-02-14 22:42:15,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:15,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-14 22:42:15,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30546.18 MB 2025-02-14 22:42:15,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12908.80 MB 2025-02-14 22:42:15,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55335.45 MB 2025-02-14 22:42:15,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35383.15 MB 2025-02-14 22:42:15,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19952.30 MB 2025-02-14 22:42:15,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30579.59 MB 2025-02-14 22:42:16,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:42:16,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:42:16,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:42:16,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:16,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30546.18 MB 2025-02-14 22:42:16,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22625.85 MB 2025-02-14 22:42:16,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7920.33 MB 2025-02-14 22:42:16,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35383.15 MB 2025-02-14 22:42:16,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35383.15 MB 2025-02-14 22:42:16,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:42:16,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33044.64 MB 2025-02-14 22:42:16,128 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 22:42:16,128 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:42:16,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:42:16,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:42:16,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:42:16,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:42:16,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22625.85 MB 2025-02-14 22:42:16,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31021.06 MB 2025-02-14 22:42:16,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 22:42:16,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35383.15 MB 2025-02-14 22:42:16,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43729.81 MB 2025-02-14 22:42:16,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 22:42:16,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31021.06 MB 2025-02-14 22:42:16,293 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 22:42:16,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:42:16,295 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:42:16,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:42:16,296 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:42:16,300 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:42:16,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:42:16,302 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:42:16,302 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:43:32,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:43:32,969 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:43:32,974 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:43:32,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:43:32,979 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2192, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:43:32,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:43:32,980 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2192, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:44:06,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:44:06,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:44:06,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.70 seconds 2025-02-14 22:44:06,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:06,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28242.91 MB 2025-02-14 22:44:06,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36000.28 MB 2025-02-14 22:44:06,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7757.37 MB 2025-02-14 22:44:06,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52076.48 MB 2025-02-14 22:44:06,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38396.76 MB 2025-02-14 22:44:06,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13679.72 MB 2025-02-14 22:44:06,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44962.85 MB 2025-02-14 22:44:06,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:44:06,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:44:06,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 22:44:06,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:06,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36000.28 MB 2025-02-14 22:44:06,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27174.42 MB 2025-02-14 22:44:06,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8825.85 MB 2025-02-14 22:44:06,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38396.76 MB 2025-02-14 22:44:06,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71665.98 MB 2025-02-14 22:44:06,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33269.22 MB 2025-02-14 22:44:06,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58746.59 MB 2025-02-14 22:44:08,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:44:08,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:44:08,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 22:44:08,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:08,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27174.42 MB 2025-02-14 22:44:08,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27705.27 MB 2025-02-14 22:44:08,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:44:08,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71665.98 MB 2025-02-14 22:44:08,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33472.64 MB 2025-02-14 22:44:08,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38193.33 MB 2025-02-14 22:44:08,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31683.81 MB 2025-02-14 22:44:08,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:44:08,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:44:08,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:44:08,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:08,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27705.27 MB 2025-02-14 22:44:08,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29594.80 MB 2025-02-14 22:44:08,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:44:08,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33472.64 MB 2025-02-14 22:44:08,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34416.36 MB 2025-02-14 22:44:08,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:44:08,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31012.23 MB 2025-02-14 22:44:09,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:44:09,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:44:09,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:44:09,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:09,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29594.80 MB 2025-02-14 22:44:09,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31836.66 MB 2025-02-14 22:44:09,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:44:09,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34416.36 MB 2025-02-14 22:44:09,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39606.81 MB 2025-02-14 22:44:09,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 22:44:09,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37380.94 MB 2025-02-14 22:44:09,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:44:09,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:44:09,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 22:44:09,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:09,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27705.27 MB 2025-02-14 22:44:09,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31836.66 MB 2025-02-14 22:44:09,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:44:09,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33472.64 MB 2025-02-14 22:44:09,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39606.81 MB 2025-02-14 22:44:09,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:44:09,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37380.94 MB 2025-02-14 22:44:09,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:44:09,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:44:09,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:44:09,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:09,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33370.20 MB 2025-02-14 22:44:09,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34137.20 MB 2025-02-14 22:44:09,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:44:09,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39606.81 MB 2025-02-14 22:44:09,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40022.05 MB 2025-02-14 22:44:09,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:44:09,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34844.99 MB 2025-02-14 22:44:09,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:44:09,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:44:09,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:44:09,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:09,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34550.09 MB 2025-02-14 22:44:09,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34778.85 MB 2025-02-14 22:44:09,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.76 MB 2025-02-14 22:44:09,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40022.05 MB 2025-02-14 22:44:09,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40022.05 MB 2025-02-14 22:44:09,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:44:09,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35007.82 MB 2025-02-14 22:44:09,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:44:09,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:44:09,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.39 seconds 2025-02-14 22:44:09,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:09,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20605.81 MB 2025-02-14 22:44:09,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34979.53 MB 2025-02-14 22:44:09,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14373.72 MB 2025-02-14 22:44:09,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52076.48 MB 2025-02-14 22:44:09,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40022.05 MB 2025-02-14 22:44:09,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12054.43 MB 2025-02-14 22:44:09,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35007.82 MB 2025-02-14 22:44:09,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:44:09,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:44:09,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:44:09,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:09,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34979.53 MB 2025-02-14 22:44:09,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25604.10 MB 2025-02-14 22:44:09,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9375.43 MB 2025-02-14 22:44:09,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40022.05 MB 2025-02-14 22:44:09,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40022.05 MB 2025-02-14 22:44:09,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:44:09,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37486.28 MB 2025-02-14 22:44:09,655 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 22:44:09,655 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:44:09,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:44:09,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:44:09,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:44:09,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:44:09,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25604.10 MB 2025-02-14 22:44:09,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34026.43 MB 2025-02-14 22:44:09,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 22:44:09,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40022.05 MB 2025-02-14 22:44:09,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48395.98 MB 2025-02-14 22:44:09,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 22:44:09,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34026.43 MB 2025-02-14 22:44:09,818 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 22:44:09,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:44:09,819 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:44:09,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:44:09,820 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:44:09,825 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:44:09,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:44:09,826 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:44:09,826 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:45:00,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:45:00,342 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:45:00,347 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:45:00,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:45:00,350 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1710, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:45:00,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:45:00,351 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1710, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:45:26,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:45:26,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:45:26,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.49 seconds 2025-02-14 22:45:26,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:26,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24884.26 MB 2025-02-14 22:45:26,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30936.64 MB 2025-02-14 22:45:26,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6052.38 MB 2025-02-14 22:45:26,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60955.82 MB 2025-02-14 22:45:26,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36693.87 MB 2025-02-14 22:45:26,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24261.95 MB 2025-02-14 22:45:26,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39792.26 MB 2025-02-14 22:45:26,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:45:26,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:45:26,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:45:26,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:26,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30936.64 MB 2025-02-14 22:45:26,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24667.61 MB 2025-02-14 22:45:26,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6269.03 MB 2025-02-14 22:45:26,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36693.87 MB 2025-02-14 22:45:26,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57768.15 MB 2025-02-14 22:45:26,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21074.28 MB 2025-02-14 22:45:26,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48753.57 MB 2025-02-14 22:45:28,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:45:28,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:45:28,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 22:45:28,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:28,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24667.61 MB 2025-02-14 22:45:28,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25198.45 MB 2025-02-14 22:45:28,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:45:28,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57768.15 MB 2025-02-14 22:45:28,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27883.73 MB 2025-02-14 22:45:28,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29884.42 MB 2025-02-14 22:45:28,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29178.03 MB 2025-02-14 22:45:28,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:45:28,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:45:28,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:45:28,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:28,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25198.45 MB 2025-02-14 22:45:28,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27087.98 MB 2025-02-14 22:45:28,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:45:28,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 22:45:28,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29771.17 MB 2025-02-14 22:45:28,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:45:28,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28505.41 MB 2025-02-14 22:45:29,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:45:29,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:45:29,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:45:29,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:29,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27087.98 MB 2025-02-14 22:45:29,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29329.84 MB 2025-02-14 22:45:29,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:45:29,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29771.17 MB 2025-02-14 22:45:29,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36849.06 MB 2025-02-14 22:45:29,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 22:45:29,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34874.12 MB 2025-02-14 22:45:29,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:45:29,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:45:29,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:45:29,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:29,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25198.45 MB 2025-02-14 22:45:29,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29329.84 MB 2025-02-14 22:45:29,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:45:29,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 22:45:29,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36849.06 MB 2025-02-14 22:45:29,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 22:45:29,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34874.12 MB 2025-02-14 22:45:29,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:45:29,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:45:29,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:45:29,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:29,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30863.38 MB 2025-02-14 22:45:29,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31630.38 MB 2025-02-14 22:45:29,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:45:29,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36849.06 MB 2025-02-14 22:45:29,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37262.20 MB 2025-02-14 22:45:29,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 22:45:29,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32338.17 MB 2025-02-14 22:45:29,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:45:29,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:45:29,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:45:29,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:29,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32043.27 MB 2025-02-14 22:45:29,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32271.52 MB 2025-02-14 22:45:29,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.25 MB 2025-02-14 22:45:29,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37262.20 MB 2025-02-14 22:45:29,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37262.20 MB 2025-02-14 22:45:29,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:45:29,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32492.66 MB 2025-02-14 22:45:29,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:45:29,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:45:29,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.97 seconds 2025-02-14 22:45:29,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:29,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18926.48 MB 2025-02-14 22:45:29,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32471.95 MB 2025-02-14 22:45:29,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13545.47 MB 2025-02-14 22:45:29,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60955.82 MB 2025-02-14 22:45:29,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37262.20 MB 2025-02-14 22:45:29,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23693.62 MB 2025-02-14 22:45:29,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32492.66 MB 2025-02-14 22:45:29,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:45:29,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:45:29,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:45:29,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:29,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32471.95 MB 2025-02-14 22:45:29,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23921.02 MB 2025-02-14 22:45:29,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8550.94 MB 2025-02-14 22:45:29,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37262.20 MB 2025-02-14 22:45:29,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37262.20 MB 2025-02-14 22:45:29,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:45:29,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34975.63 MB 2025-02-14 22:45:29,614 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 22:45:29,615 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:45:29,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:45:29,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:45:29,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:45:29,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:45:29,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23921.02 MB 2025-02-14 22:45:29,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32333.45 MB 2025-02-14 22:45:29,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 22:45:29,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37262.20 MB 2025-02-14 22:45:29,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45625.64 MB 2025-02-14 22:45:29,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 22:45:29,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32333.45 MB 2025-02-14 22:45:29,780 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 22:45:29,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:45:29,782 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:45:29,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:45:29,782 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:45:29,787 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:45:29,788 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:45:29,788 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:45:29,788 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:46:14,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:14,893 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:46:14,898 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:46:14,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:14,902 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1324, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:46:14,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:14,903 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1324, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:46:35,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:46:35,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:46:35,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.45 seconds 2025-02-14 22:46:35,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:35,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.55 MB 2025-02-14 22:46:35,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26880.11 MB 2025-02-14 22:46:35,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4685.56 MB 2025-02-14 22:46:35,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53989.08 MB 2025-02-14 22:46:35,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35334.91 MB 2025-02-14 22:46:35,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18654.17 MB 2025-02-14 22:46:35,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35742.79 MB 2025-02-14 22:46:35,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:46:35,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:46:35,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 22:46:35,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:35,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26880.11 MB 2025-02-14 22:46:35,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22661.96 MB 2025-02-14 22:46:35,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4218.15 MB 2025-02-14 22:46:35,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35334.91 MB 2025-02-14 22:46:35,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37459.33 MB 2025-02-14 22:46:35,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2124.41 MB 2025-02-14 22:46:35,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34231.52 MB 2025-02-14 22:46:37,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:46:37,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:46:37,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:46:37,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:37,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22661.96 MB 2025-02-14 22:46:37,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23192.80 MB 2025-02-14 22:46:37,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:46:37,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37459.33 MB 2025-02-14 22:46:37,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32772.19 MB 2025-02-14 22:46:37,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4687.13 MB 2025-02-14 22:46:37,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27171.60 MB 2025-02-14 22:46:37,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:46:37,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:46:37,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:46:37,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:37,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23192.80 MB 2025-02-14 22:46:37,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25082.34 MB 2025-02-14 22:46:37,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:46:37,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32772.19 MB 2025-02-14 22:46:37,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32772.19 MB 2025-02-14 22:46:37,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:46:37,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26499.77 MB 2025-02-14 22:46:37,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:46:37,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:46:37,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:46:37,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:37,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25082.34 MB 2025-02-14 22:46:37,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27324.19 MB 2025-02-14 22:46:37,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:46:37,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32772.19 MB 2025-02-14 22:46:37,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35131.49 MB 2025-02-14 22:46:37,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 22:46:37,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32868.48 MB 2025-02-14 22:46:37,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:46:37,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:46:37,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:46:37,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:37,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23192.80 MB 2025-02-14 22:46:37,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27324.19 MB 2025-02-14 22:46:37,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:46:37,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32772.19 MB 2025-02-14 22:46:37,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35131.49 MB 2025-02-14 22:46:37,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 22:46:37,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32868.48 MB 2025-02-14 22:46:37,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:46:37,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:46:37,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:46:37,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:37,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28857.74 MB 2025-02-14 22:46:37,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29624.74 MB 2025-02-14 22:46:37,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:46:37,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35131.49 MB 2025-02-14 22:46:37,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35546.73 MB 2025-02-14 22:46:37,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:46:37,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30332.53 MB 2025-02-14 22:46:37,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:46:37,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:46:37,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:46:37,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:37,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30037.63 MB 2025-02-14 22:46:37,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30273.19 MB 2025-02-14 22:46:37,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 235.56 MB 2025-02-14 22:46:37,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35546.73 MB 2025-02-14 22:46:37,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 22:46:37,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 22:46:37,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30421.09 MB 2025-02-14 22:46:37,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:46:37,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:46:37,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.85 seconds 2025-02-14 22:46:37,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:37,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17581.63 MB 2025-02-14 22:46:37,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30474.26 MB 2025-02-14 22:46:37,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12892.63 MB 2025-02-14 22:46:37,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53989.08 MB 2025-02-14 22:46:37,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 22:46:37,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18440.26 MB 2025-02-14 22:46:37,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30474.26 MB 2025-02-14 22:46:38,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:46:38,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:46:38,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:46:38,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:38,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30474.26 MB 2025-02-14 22:46:38,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22586.02 MB 2025-02-14 22:46:38,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7888.24 MB 2025-02-14 22:46:38,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35548.82 MB 2025-02-14 22:46:38,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 22:46:38,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:46:38,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.93 MB 2025-02-14 22:46:38,043 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:46:38,043 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:46:38,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:46:38,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:46:38,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:46:38,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:46:38,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22586.02 MB 2025-02-14 22:46:38,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31025.04 MB 2025-02-14 22:46:38,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:46:38,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35548.82 MB 2025-02-14 22:46:38,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43939.53 MB 2025-02-14 22:46:38,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:46:38,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31025.04 MB 2025-02-14 22:46:38,212 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:46:38,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:38,214 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:46:38,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:38,215 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:46:38,219 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:46:38,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:38,220 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:46:38,221 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:46:49,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:49,151 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:46:49,156 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:46:49,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:49,160 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1053, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:46:49,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:46:49,161 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1053, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:47:05,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:47:05,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:47:05,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.42 seconds 2025-02-14 22:47:05,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:05,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20306.18 MB 2025-02-14 22:47:05,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24032.69 MB 2025-02-14 22:47:05,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3726.51 MB 2025-02-14 22:47:05,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56524.54 MB 2025-02-14 22:47:05,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26470.25 MB 2025-02-14 22:47:05,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30054.29 MB 2025-02-14 22:47:05,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32948.44 MB 2025-02-14 22:47:05,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:47:05,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:47:05,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:47:05,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:05,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24032.69 MB 2025-02-14 22:47:05,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21253.12 MB 2025-02-14 22:47:05,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2779.57 MB 2025-02-14 22:47:05,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26470.25 MB 2025-02-14 22:47:05,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44033.90 MB 2025-02-14 22:47:05,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17563.65 MB 2025-02-14 22:47:05,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35622.67 MB 2025-02-14 22:47:07,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:47:07,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:47:07,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 22:47:07,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:07,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21253.12 MB 2025-02-14 22:47:07,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21783.96 MB 2025-02-14 22:47:07,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:47:07,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44033.90 MB 2025-02-14 22:47:07,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29303.50 MB 2025-02-14 22:47:07,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14730.40 MB 2025-02-14 22:47:07,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25762.51 MB 2025-02-14 22:47:07,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:47:07,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:47:07,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:47:07,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:07,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.96 MB 2025-02-14 22:47:07,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23673.49 MB 2025-02-14 22:47:07,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:47:07,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29303.50 MB 2025-02-14 22:47:07,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29303.50 MB 2025-02-14 22:47:07,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:47:07,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25090.92 MB 2025-02-14 22:47:07,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:47:07,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:47:07,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:47:07,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:07,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23673.49 MB 2025-02-14 22:47:07,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25915.35 MB 2025-02-14 22:47:07,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:47:07,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29303.50 MB 2025-02-14 22:47:07,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-14 22:47:07,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 22:47:07,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31459.63 MB 2025-02-14 22:47:07,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:47:07,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:47:07,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:47:07,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:07,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.96 MB 2025-02-14 22:47:07,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25915.35 MB 2025-02-14 22:47:07,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:47:07,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29303.50 MB 2025-02-14 22:47:07,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33078.38 MB 2025-02-14 22:47:07,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 22:47:07,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31459.63 MB 2025-02-14 22:47:07,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:47:07,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:47:07,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:47:07,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:07,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27448.89 MB 2025-02-14 22:47:07,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28215.89 MB 2025-02-14 22:47:07,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:47:07,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33078.38 MB 2025-02-14 22:47:07,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-14 22:47:07,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:47:07,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28923.68 MB 2025-02-14 22:47:07,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:47:08,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:47:08,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:47:08,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:08,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28628.78 MB 2025-02-14 22:47:08,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28856.86 MB 2025-02-14 22:47:08,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 22:47:08,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33493.61 MB 2025-02-14 22:47:08,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-14 22:47:08,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:47:08,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29095.78 MB 2025-02-14 22:47:08,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:47:08,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:47:08,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.84 seconds 2025-02-14 22:47:08,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:08,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16637.44 MB 2025-02-14 22:47:08,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29057.78 MB 2025-02-14 22:47:08,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12420.34 MB 2025-02-14 22:47:08,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56524.54 MB 2025-02-14 22:47:08,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-14 22:47:08,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23030.92 MB 2025-02-14 22:47:08,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29095.78 MB 2025-02-14 22:47:08,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:47:08,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:47:08,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:47:08,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:08,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29057.78 MB 2025-02-14 22:47:08,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21639.55 MB 2025-02-14 22:47:08,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7418.24 MB 2025-02-14 22:47:08,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33493.61 MB 2025-02-14 22:47:08,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33493.61 MB 2025-02-14 22:47:08,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:47:08,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.61 MB 2025-02-14 22:47:08,289 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 22:47:08,289 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:47:08,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:47:08,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:47:08,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:47:08,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:47:08,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21639.55 MB 2025-02-14 22:47:08,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30072.84 MB 2025-02-14 22:47:08,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 22:47:08,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33493.61 MB 2025-02-14 22:47:08,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41878.03 MB 2025-02-14 22:47:08,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 22:47:08,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30072.84 MB 2025-02-14 22:47:08,452 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 22:47:08,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:47:08,453 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:47:08,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:47:08,454 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:47:08,459 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:47:08,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:47:08,460 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:47:08,460 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:48:14,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:14,960 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:48:14,965 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:48:14,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:14,969 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 229, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:48:14,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:14,969 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 229, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:48:18,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:48:18,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:48:18,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.53 seconds 2025-02-14 22:48:18,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:18,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14564.42 MB 2025-02-14 22:48:18,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15374.83 MB 2025-02-14 22:48:18,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 810.42 MB 2025-02-14 22:48:18,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50262.44 MB 2025-02-14 22:48:18,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:48:18,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26984.05 MB 2025-02-14 22:48:18,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24262.28 MB 2025-02-14 22:48:18,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:48:18,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:48:18,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:48:18,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:18,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15374.83 MB 2025-02-14 22:48:18,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15619.93 MB 2025-02-14 22:48:18,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.10 MB 2025-02-14 22:48:18,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:48:18,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23278.39 MB 2025-02-14 22:48:18,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:48:18,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18314.11 MB 2025-02-14 22:48:19,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:48:19,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:48:19,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.99 seconds 2025-02-14 22:48:19,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:19,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15619.93 MB 2025-02-14 22:48:19,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15895.97 MB 2025-02-14 22:48:19,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-14 22:48:19,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23278.39 MB 2025-02-14 22:48:19,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 22:48:19,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 22:48:19,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19874.51 MB 2025-02-14 22:48:19,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:48:19,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:48:19,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:48:19,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:19,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15895.97 MB 2025-02-14 22:48:19,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16878.29 MB 2025-02-14 22:48:19,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.32 MB 2025-02-14 22:48:19,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 22:48:19,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 22:48:19,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:48:19,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17615.35 MB 2025-02-14 22:48:19,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:48:19,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:48:19,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:48:19,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:19,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16878.29 MB 2025-02-14 22:48:19,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18044.08 MB 2025-02-14 22:48:19,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1165.80 MB 2025-02-14 22:48:19,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 22:48:19,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 22:48:19,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:48:19,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20927.08 MB 2025-02-14 22:48:19,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:48:19,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:48:19,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 22:48:19,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:19,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15895.97 MB 2025-02-14 22:48:19,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18044.08 MB 2025-02-14 22:48:19,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2148.11 MB 2025-02-14 22:48:19,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 22:48:19,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 22:48:19,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:48:19,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20927.08 MB 2025-02-14 22:48:19,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:48:19,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:48:19,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:48:19,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:19,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18841.52 MB 2025-02-14 22:48:19,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19240.37 MB 2025-02-14 22:48:19,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 398.84 MB 2025-02-14 22:48:19,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 22:48:19,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23020.44 MB 2025-02-14 22:48:19,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-14 22:48:19,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19610.39 MB 2025-02-14 22:48:19,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:48:19,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:48:19,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:48:19,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:19,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19455.07 MB 2025-02-14 22:48:19,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19683.76 MB 2025-02-14 22:48:19,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-14 22:48:19,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23020.44 MB 2025-02-14 22:48:19,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23020.44 MB 2025-02-14 22:48:19,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:48:19,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19725.35 MB 2025-02-14 22:48:19,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:48:19,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:48:19,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.77 seconds 2025-02-14 22:48:19,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:19,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13766.56 MB 2025-02-14 22:48:19,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19884.39 MB 2025-02-14 22:48:19,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6117.83 MB 2025-02-14 22:48:19,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50262.44 MB 2025-02-14 22:48:19,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23020.44 MB 2025-02-14 22:48:19,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27242.00 MB 2025-02-14 22:48:19,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19884.39 MB 2025-02-14 22:48:20,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:48:20,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:48:20,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:48:20,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:20,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14850.59 MB 2025-02-14 22:48:20,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17857.99 MB 2025-02-14 22:48:20,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3007.40 MB 2025-02-14 22:48:20,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23020.44 MB 2025-02-14 22:48:20,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23020.44 MB 2025-02-14 22:48:20,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:48:20,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18158.69 MB 2025-02-14 22:48:20,022 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 22:48:20,022 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:48:20,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:48:20,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:48:20,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:48:20,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:48:20,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17857.99 MB 2025-02-14 22:48:20,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26278.77 MB 2025-02-14 22:48:20,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 22:48:20,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23020.44 MB 2025-02-14 22:48:20,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31392.27 MB 2025-02-14 22:48:20,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 22:48:20,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26278.77 MB 2025-02-14 22:48:20,184 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 22:48:20,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:20,185 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:48:20,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:20,186 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:48:20,191 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:48:20,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:20,192 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:48:20,192 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:48:42,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:42,858 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:48:42,863 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:48:42,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:42,866 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1502, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:48:42,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:48:42,867 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1502, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:49:06,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:49:06,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:49:06,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.15 seconds 2025-02-14 22:49:06,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:06,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23434.88 MB 2025-02-14 22:49:06,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28751.16 MB 2025-02-14 22:49:06,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5316.28 MB 2025-02-14 22:49:06,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39764.10 MB 2025-02-14 22:49:06,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35079.06 MB 2025-02-14 22:49:06,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4685.04 MB 2025-02-14 22:49:06,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37663.40 MB 2025-02-14 22:49:06,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:49:06,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:49:06,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:49:06,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:06,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28751.16 MB 2025-02-14 22:49:06,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23586.28 MB 2025-02-14 22:49:06,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5164.88 MB 2025-02-14 22:49:06,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35079.06 MB 2025-02-14 22:49:06,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49117.40 MB 2025-02-14 22:49:06,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14038.34 MB 2025-02-14 22:49:06,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42836.73 MB 2025-02-14 22:49:08,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:49:08,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:49:08,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 22:49:08,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23586.28 MB 2025-02-14 22:49:08,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24117.12 MB 2025-02-14 22:49:08,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:49:08,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49117.40 MB 2025-02-14 22:49:08,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26992.44 MB 2025-02-14 22:49:08,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22124.95 MB 2025-02-14 22:49:08,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28095.67 MB 2025-02-14 22:49:08,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:49:08,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:49:08,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:49:08,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24117.12 MB 2025-02-14 22:49:08,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26006.66 MB 2025-02-14 22:49:08,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:49:08,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26992.44 MB 2025-02-14 22:49:08,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29823.60 MB 2025-02-14 22:49:08,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 22:49:08,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27424.08 MB 2025-02-14 22:49:08,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:49:08,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:49:08,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:49:08,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26006.66 MB 2025-02-14 22:49:08,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28248.51 MB 2025-02-14 22:49:08,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:49:08,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29823.60 MB 2025-02-14 22:49:08,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35957.77 MB 2025-02-14 22:49:08,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:49:08,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33792.79 MB 2025-02-14 22:49:08,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:49:08,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:49:08,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:49:08,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24117.12 MB 2025-02-14 22:49:08,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28248.51 MB 2025-02-14 22:49:08,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:49:08,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26992.44 MB 2025-02-14 22:49:08,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35957.77 MB 2025-02-14 22:49:08,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 22:49:08,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33792.79 MB 2025-02-14 22:49:08,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:49:08,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:49:08,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 22:49:08,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29782.05 MB 2025-02-14 22:49:08,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30549.06 MB 2025-02-14 22:49:08,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:49:08,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35957.77 MB 2025-02-14 22:49:08,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36373.00 MB 2025-02-14 22:49:08,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:49:08,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31256.84 MB 2025-02-14 22:49:08,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:49:08,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:49:08,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:49:08,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30961.95 MB 2025-02-14 22:49:08,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31191.86 MB 2025-02-14 22:49:08,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.92 MB 2025-02-14 22:49:08,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36373.00 MB 2025-02-14 22:49:08,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36373.00 MB 2025-02-14 22:49:08,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:49:08,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31390.33 MB 2025-02-14 22:49:08,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:49:08,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:49:08,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.60 seconds 2025-02-14 22:49:08,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18201.79 MB 2025-02-14 22:49:08,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31392.94 MB 2025-02-14 22:49:08,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13191.14 MB 2025-02-14 22:49:08,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39764.10 MB 2025-02-14 22:49:08,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36373.00 MB 2025-02-14 22:49:08,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3391.09 MB 2025-02-14 22:49:08,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31392.94 MB 2025-02-14 22:49:08,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:49:08,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:49:08,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:49:08,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31392.94 MB 2025-02-14 22:49:08,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23206.18 MB 2025-02-14 22:49:08,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8186.75 MB 2025-02-14 22:49:08,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36373.00 MB 2025-02-14 22:49:08,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36373.00 MB 2025-02-14 22:49:08,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:49:08,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33904.60 MB 2025-02-14 22:49:08,762 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:49:08,763 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:49:08,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:49:08,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:49:08,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:49:08,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:49:08,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23206.18 MB 2025-02-14 22:49:08,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31645.21 MB 2025-02-14 22:49:08,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:49:08,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36373.00 MB 2025-02-14 22:49:08,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44763.71 MB 2025-02-14 22:49:08,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:49:08,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31645.21 MB 2025-02-14 22:49:08,934 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:49:08,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:49:08,936 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:49:08,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:49:08,937 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:49:08,942 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:49:08,943 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:49:08,943 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:49:08,943 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:50:45,106 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:50:45,106 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:50:45,111 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:50:45,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:50:45,116 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 468, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:50:45,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:50:45,118 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 468, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:50:52,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:50:52,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:50:52,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.20 seconds 2025-02-14 22:50:52,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:52,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16229.81 MB 2025-02-14 22:50:52,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17886.03 MB 2025-02-14 22:50:52,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1656.23 MB 2025-02-14 22:50:52,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57348.72 MB 2025-02-14 22:50:52,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25167.92 MB 2025-02-14 22:50:52,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32180.80 MB 2025-02-14 22:50:52,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26833.64 MB 2025-02-14 22:50:52,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:50:52,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:50:52,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 22:50:52,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:52,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17886.03 MB 2025-02-14 22:50:52,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18211.89 MB 2025-02-14 22:50:52,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 325.85 MB 2025-02-14 22:50:52,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25167.92 MB 2025-02-14 22:50:52,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28707.91 MB 2025-02-14 22:50:52,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3539.99 MB 2025-02-14 22:50:52,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25237.22 MB 2025-02-14 22:50:54,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:50:54,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:50:54,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 22:50:54,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18211.89 MB 2025-02-14 22:50:54,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18742.73 MB 2025-02-14 22:50:54,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:50:54,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28707.91 MB 2025-02-14 22:50:54,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24488.44 MB 2025-02-14 22:50:54,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4219.47 MB 2025-02-14 22:50:54,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22721.27 MB 2025-02-14 22:50:54,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:50:54,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:50:54,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:50:54,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.73 MB 2025-02-14 22:50:54,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20632.26 MB 2025-02-14 22:50:54,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:50:54,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24488.44 MB 2025-02-14 22:50:54,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24488.44 MB 2025-02-14 22:50:54,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:50:54,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22049.69 MB 2025-02-14 22:50:54,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:50:54,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:50:54,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:50:54,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20632.26 MB 2025-02-14 22:50:54,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22874.12 MB 2025-02-14 22:50:54,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:50:54,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24488.44 MB 2025-02-14 22:50:54,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30622.61 MB 2025-02-14 22:50:54,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:50:54,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28418.40 MB 2025-02-14 22:50:54,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:50:54,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:50:54,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:50:54,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.73 MB 2025-02-14 22:50:54,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22874.12 MB 2025-02-14 22:50:54,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:50:54,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24488.44 MB 2025-02-14 22:50:54,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30622.61 MB 2025-02-14 22:50:54,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:50:54,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28418.40 MB 2025-02-14 22:50:54,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:50:54,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:50:54,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:50:54,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24407.66 MB 2025-02-14 22:50:54,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25174.66 MB 2025-02-14 22:50:54,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:50:54,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30622.61 MB 2025-02-14 22:50:54,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31035.75 MB 2025-02-14 22:50:54,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 22:50:54,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25882.45 MB 2025-02-14 22:50:54,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:50:54,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:50:54,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:50:54,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25587.55 MB 2025-02-14 22:50:54,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25819.29 MB 2025-02-14 22:50:54,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.74 MB 2025-02-14 22:50:54,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31035.75 MB 2025-02-14 22:50:54,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31035.75 MB 2025-02-14 22:50:54,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:50:54,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26015.52 MB 2025-02-14 22:50:54,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:50:54,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:50:54,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.57 seconds 2025-02-14 22:50:54,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.26 MB 2025-02-14 22:50:54,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26020.36 MB 2025-02-14 22:50:54,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11421.11 MB 2025-02-14 22:50:54,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57348.72 MB 2025-02-14 22:50:54,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31035.75 MB 2025-02-14 22:50:54,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26312.97 MB 2025-02-14 22:50:54,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26020.36 MB 2025-02-14 22:50:54,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:50:54,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:50:54,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:50:54,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26020.36 MB 2025-02-14 22:50:54,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19603.64 MB 2025-02-14 22:50:54,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6416.72 MB 2025-02-14 22:50:54,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31035.75 MB 2025-02-14 22:50:54,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31035.75 MB 2025-02-14 22:50:54,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:50:54,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28532.03 MB 2025-02-14 22:50:54,977 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:50:54,977 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:50:54,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:50:54,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:50:54,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:50:54,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:50:54,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19603.64 MB 2025-02-14 22:50:54,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28042.67 MB 2025-02-14 22:50:54,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:50:54,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31035.75 MB 2025-02-14 22:50:54,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39426.46 MB 2025-02-14 22:50:54,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:50:54,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28042.67 MB 2025-02-14 22:50:55,139 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:50:55,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:50:55,141 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:50:55,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:50:55,142 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:50:55,146 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:50:55,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:50:55,147 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:50:55,147 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:51:04,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:51:04,295 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:51:04,301 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:51:04,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:51:04,305 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2373, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:51:04,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:51:04,306 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2373, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:51:40,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:51:40,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:51:40,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.66 seconds 2025-02-14 22:51:40,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:40,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29504.15 MB 2025-02-14 22:51:40,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37902.06 MB 2025-02-14 22:51:40,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.91 MB 2025-02-14 22:51:40,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60412.66 MB 2025-02-14 22:51:40,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43306.19 MB 2025-02-14 22:51:40,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17106.47 MB 2025-02-14 22:51:40,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46902.76 MB 2025-02-14 22:51:41,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:51:41,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:51:41,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 22:51:41,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:41,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37902.06 MB 2025-02-14 22:51:41,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28114.34 MB 2025-02-14 22:51:41,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9787.73 MB 2025-02-14 22:51:41,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43306.19 MB 2025-02-14 22:51:41,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 73054.29 MB 2025-02-14 22:51:41,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29748.10 MB 2025-02-14 22:51:41,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62881.06 MB 2025-02-14 22:51:43,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:51:43,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:51:43,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 22:51:43,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28114.34 MB 2025-02-14 22:51:43,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28645.18 MB 2025-02-14 22:51:43,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:51:43,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73054.29 MB 2025-02-14 22:51:43,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32122.08 MB 2025-02-14 22:51:43,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40932.21 MB 2025-02-14 22:51:43,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32624.76 MB 2025-02-14 22:51:43,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:51:43,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:51:43,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:51:43,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28645.18 MB 2025-02-14 22:51:43,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30534.25 MB 2025-02-14 22:51:43,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.08 MB 2025-02-14 22:51:43,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32122.08 MB 2025-02-14 22:51:43,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34009.51 MB 2025-02-14 22:51:43,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:51:43,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31951.68 MB 2025-02-14 22:51:43,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:51:43,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:51:43,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:51:43,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30534.25 MB 2025-02-14 22:51:43,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32776.11 MB 2025-02-14 22:51:43,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:51:43,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34009.51 MB 2025-02-14 22:51:43,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40143.68 MB 2025-02-14 22:51:43,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 22:51:43,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38320.39 MB 2025-02-14 22:51:43,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:51:43,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:51:43,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:51:43,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28645.18 MB 2025-02-14 22:51:43,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32776.11 MB 2025-02-14 22:51:43,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.93 MB 2025-02-14 22:51:43,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32122.08 MB 2025-02-14 22:51:43,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40143.68 MB 2025-02-14 22:51:43,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 22:51:43,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38320.39 MB 2025-02-14 22:51:43,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:51:43,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:51:43,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:51:43,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34309.65 MB 2025-02-14 22:51:43,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35076.65 MB 2025-02-14 22:51:43,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:51:43,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40143.68 MB 2025-02-14 22:51:43,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40556.82 MB 2025-02-14 22:51:43,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 22:51:43,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35784.44 MB 2025-02-14 22:51:43,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:51:43,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:51:43,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:51:43,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35489.54 MB 2025-02-14 22:51:43,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35717.79 MB 2025-02-14 22:51:43,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.25 MB 2025-02-14 22:51:43,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40556.82 MB 2025-02-14 22:51:43,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40556.82 MB 2025-02-14 22:51:43,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:51:43,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35960.39 MB 2025-02-14 22:51:43,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:51:43,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:51:43,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.22 seconds 2025-02-14 22:51:43,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21236.43 MB 2025-02-14 22:51:43,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35918.23 MB 2025-02-14 22:51:43,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14681.80 MB 2025-02-14 22:51:43,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56212.06 MB 2025-02-14 22:51:43,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40556.82 MB 2025-02-14 22:51:43,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15655.24 MB 2025-02-14 22:51:43,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35960.39 MB 2025-02-14 22:51:43,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:51:43,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:51:43,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:51:43,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35918.23 MB 2025-02-14 22:51:43,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26230.96 MB 2025-02-14 22:51:43,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9687.26 MB 2025-02-14 22:51:43,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40556.82 MB 2025-02-14 22:51:43,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40556.82 MB 2025-02-14 22:51:43,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:51:43,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38421.95 MB 2025-02-14 22:51:43,816 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 22:51:43,817 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 22:51:43,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:51:43,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:51:43,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:51:43,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:51:43,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26230.96 MB 2025-02-14 22:51:43,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34643.39 MB 2025-02-14 22:51:43,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 22:51:43,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40556.82 MB 2025-02-14 22:51:43,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48920.26 MB 2025-02-14 22:51:43,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 22:51:43,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34643.39 MB 2025-02-14 22:51:43,985 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 22:51:43,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:51:43,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:51:43,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:51:43,988 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:51:43,993 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:51:43,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:51:43,994 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:51:43,994 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 22:52:59,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:52:59,383 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:52:59,388 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:52:59,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:52:59,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:52:59,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:52:59,395 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:53:01,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:53:01,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:53:01,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.13 seconds 2025-02-14 22:53:01,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:01,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13916.38 MB 2025-02-14 22:53:01,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14397.67 MB 2025-02-14 22:53:01,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.30 MB 2025-02-14 22:53:01,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57283.71 MB 2025-02-14 22:53:01,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:53:01,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37752.93 MB 2025-02-14 22:53:01,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23387.75 MB 2025-02-14 22:53:01,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:53:01,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:53:01,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:53:01,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:01,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14397.67 MB 2025-02-14 22:53:01,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14588.72 MB 2025-02-14 22:53:01,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.05 MB 2025-02-14 22:53:01,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:53:01,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:53:01,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:01,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16252.04 MB 2025-02-14 22:53:02,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:53:02,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:53:02,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.63 seconds 2025-02-14 22:53:02,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14588.72 MB 2025-02-14 22:53:02,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14761.24 MB 2025-02-14 22:53:02,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-14 22:53:02,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:53:02,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:53:02,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:02,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18758.37 MB 2025-02-14 22:53:02,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:53:02,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:53:02,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:53:02,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-14 22:53:02,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15375.13 MB 2025-02-14 22:53:02,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-14 22:53:02,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:53:02,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:53:02,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:02,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15835.80 MB 2025-02-14 22:53:02,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:53:02,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:53:02,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:53:02,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15375.13 MB 2025-02-14 22:53:02,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-14 22:53:02,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-14 22:53:02,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:53:02,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:53:02,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:02,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-14 22:53:02,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:53:02,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:53:02,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:53:02,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-14 22:53:02,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-14 22:53:02,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-14 22:53:02,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:53:02,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:53:02,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:02,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-14 22:53:02,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:53:02,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:53:02,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 22:53:02,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16602.18 MB 2025-02-14 22:53:02,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16851.45 MB 2025-02-14 22:53:02,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-14 22:53:02,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:53:02,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19662.90 MB 2025-02-14 22:53:02,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 22:53:02,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17094.01 MB 2025-02-14 22:53:02,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:53:02,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:53:02,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:53:02,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16985.65 MB 2025-02-14 22:53:02,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17190.96 MB 2025-02-14 22:53:02,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.31 MB 2025-02-14 22:53:02,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19662.90 MB 2025-02-14 22:53:02,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-14 22:53:02,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 22:53:02,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17204.65 MB 2025-02-14 22:53:02,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:53:02,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:53:02,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.93 seconds 2025-02-14 22:53:02,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13442.54 MB 2025-02-14 22:53:02,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17391.93 MB 2025-02-14 22:53:02,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3949.39 MB 2025-02-14 22:53:02,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57283.71 MB 2025-02-14 22:53:02,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-14 22:53:02,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37616.62 MB 2025-02-14 22:53:02,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17391.93 MB 2025-02-14 22:53:02,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:53:02,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:53:02,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:53:02,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17391.93 MB 2025-02-14 22:53:02,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17171.20 MB 2025-02-14 22:53:02,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -220.73 MB 2025-02-14 22:53:02,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19667.09 MB 2025-02-14 22:53:02,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19667.09 MB 2025-02-14 22:53:02,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:02,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18797.78 MB 2025-02-14 22:53:02,609 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 22:53:02,609 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:53:02,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:53:02,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:53:02,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:53:02,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:02,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17171.20 MB 2025-02-14 22:53:02,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25606.05 MB 2025-02-14 22:53:02,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 22:53:02,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19667.09 MB 2025-02-14 22:53:02,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30150.75 MB 2025-02-14 22:53:02,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10483.66 MB 2025-02-14 22:53:02,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25606.05 MB 2025-02-14 22:53:02,776 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 22:53:02,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:02,778 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:53:02,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:02,779 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:53:02,784 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:53:02,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:02,785 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:53:02,785 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:53:25,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:25,948 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:53:25,953 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:53:25,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:25,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1727, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:53:25,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:25,957 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1727, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:53:52,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:53:52,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:53:52,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.63 seconds 2025-02-14 22:53:52,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:52,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25002.72 MB 2025-02-14 22:53:52,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31114.48 MB 2025-02-14 22:53:52,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6111.76 MB 2025-02-14 22:53:52,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42729.47 MB 2025-02-14 22:53:52,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36805.02 MB 2025-02-14 22:53:52,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5924.45 MB 2025-02-14 22:53:52,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39992.64 MB 2025-02-14 22:53:52,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:53:52,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:53:52,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 22:53:52,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:52,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31114.48 MB 2025-02-14 22:53:52,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24755.99 MB 2025-02-14 22:53:52,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6358.49 MB 2025-02-14 22:53:52,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36805.02 MB 2025-02-14 22:53:52,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57371.79 MB 2025-02-14 22:53:52,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20566.77 MB 2025-02-14 22:53:52,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48523.10 MB 2025-02-14 22:53:54,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:53:54,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:53:54,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 22:53:54,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:54,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24755.99 MB 2025-02-14 22:53:54,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25286.83 MB 2025-02-14 22:53:54,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:53:54,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57371.79 MB 2025-02-14 22:53:54,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32107.40 MB 2025-02-14 22:53:54,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25264.39 MB 2025-02-14 22:53:54,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29265.37 MB 2025-02-14 22:53:54,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:53:54,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:53:54,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:53:54,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:54,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25286.83 MB 2025-02-14 22:53:54,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27176.36 MB 2025-02-14 22:53:54,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:53:54,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32107.40 MB 2025-02-14 22:53:54,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32107.40 MB 2025-02-14 22:53:54,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:54,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28593.79 MB 2025-02-14 22:53:54,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:53:54,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:53:54,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:53:54,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:54,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27176.36 MB 2025-02-14 22:53:54,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29418.22 MB 2025-02-14 22:53:54,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:53:54,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32107.40 MB 2025-02-14 22:53:54,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36825.99 MB 2025-02-14 22:53:54,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 22:53:54,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34962.50 MB 2025-02-14 22:53:54,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:53:54,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:53:54,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:53:54,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:54,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25286.83 MB 2025-02-14 22:53:54,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29418.22 MB 2025-02-14 22:53:54,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:53:54,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32107.40 MB 2025-02-14 22:53:54,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36825.99 MB 2025-02-14 22:53:54,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 22:53:54,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34962.50 MB 2025-02-14 22:53:55,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:53:55,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:53:55,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-14 22:53:55,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:55,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30951.76 MB 2025-02-14 22:53:55,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31718.76 MB 2025-02-14 22:53:55,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:53:55,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36825.99 MB 2025-02-14 22:53:55,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37241.23 MB 2025-02-14 22:53:55,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:53:55,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32426.55 MB 2025-02-14 22:53:55,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:53:55,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:53:55,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:53:55,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:55,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32131.65 MB 2025-02-14 22:53:55,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32360.61 MB 2025-02-14 22:53:55,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 22:53:55,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37241.23 MB 2025-02-14 22:53:55,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37241.23 MB 2025-02-14 22:53:55,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:55,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32603.15 MB 2025-02-14 22:53:55,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:53:55,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:53:55,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.27 seconds 2025-02-14 22:53:55,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:55,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18985.71 MB 2025-02-14 22:53:55,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32561.49 MB 2025-02-14 22:53:55,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13575.77 MB 2025-02-14 22:53:55,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42729.47 MB 2025-02-14 22:53:55,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37241.23 MB 2025-02-14 22:53:55,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5488.25 MB 2025-02-14 22:53:55,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32603.15 MB 2025-02-14 22:53:55,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:53:55,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:53:55,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:53:55,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:55,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32561.49 MB 2025-02-14 22:53:55,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23987.05 MB 2025-02-14 22:53:55,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8574.43 MB 2025-02-14 22:53:55,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37241.23 MB 2025-02-14 22:53:55,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37241.23 MB 2025-02-14 22:53:55,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:53:55,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32561.49 MB 2025-02-14 22:53:55,514 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 22:53:55,514 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 22:53:55,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:53:55,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:53:55,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:53:55,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:53:55,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23987.05 MB 2025-02-14 22:53:55,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32417.73 MB 2025-02-14 22:53:55,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 22:53:55,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37241.23 MB 2025-02-14 22:53:55,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41431.33 MB 2025-02-14 22:53:55,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 22:53:55,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32417.73 MB 2025-02-14 22:53:55,677 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 22:53:55,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:55,678 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:53:55,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:55,679 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:53:55,684 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:53:55,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:53:55,685 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:53:55,685 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 22:54:04,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:54:04,947 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:54:04,952 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:54:04,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:54:04,955 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 519, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:54:04,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:54:04,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 519, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:54:13,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:54:13,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:54:13,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.07 seconds 2025-02-14 22:54:13,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:13,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28181.37 MB 2025-02-14 22:54:13,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30018.08 MB 2025-02-14 22:54:13,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1836.71 MB 2025-02-14 22:54:13,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54003.76 MB 2025-02-14 22:54:13,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35412.51 MB 2025-02-14 22:54:13,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18591.25 MB 2025-02-14 22:54:13,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39011.69 MB 2025-02-14 22:54:13,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:54:13,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:54:13,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 22:54:13,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:13,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30018.08 MB 2025-02-14 22:54:13,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30073.20 MB 2025-02-14 22:54:13,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 55.12 MB 2025-02-14 22:54:13,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35412.51 MB 2025-02-14 22:54:13,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42138.08 MB 2025-02-14 22:54:13,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6725.57 MB 2025-02-14 22:54:13,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37606.39 MB 2025-02-14 22:54:14,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:54:14,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:54:14,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 22:54:14,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:14,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30073.20 MB 2025-02-14 22:54:14,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30604.05 MB 2025-02-14 22:54:14,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:54:14,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42138.08 MB 2025-02-14 22:54:14,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36593.21 MB 2025-02-14 22:54:14,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5544.87 MB 2025-02-14 22:54:14,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34582.59 MB 2025-02-14 22:54:14,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:54:14,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:54:14,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:54:14,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:14,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30604.05 MB 2025-02-14 22:54:14,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32493.31 MB 2025-02-14 22:54:14,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.26 MB 2025-02-14 22:54:14,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36593.21 MB 2025-02-14 22:54:14,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37536.92 MB 2025-02-14 22:54:14,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:54:14,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33910.74 MB 2025-02-14 22:54:15,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:54:15,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:54:15,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:54:15,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:15,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32493.31 MB 2025-02-14 22:54:15,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23138.98 MB 2025-02-14 22:54:15,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9354.33 MB 2025-02-14 22:54:15,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37536.92 MB 2025-02-14 22:54:15,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37536.92 MB 2025-02-14 22:54:15,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:54:15,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34619.59 MB 2025-02-14 22:54:15,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:54:15,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:54:15,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 22:54:15,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:15,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30604.05 MB 2025-02-14 22:54:15,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23138.98 MB 2025-02-14 22:54:15,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7465.07 MB 2025-02-14 22:54:15,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36593.21 MB 2025-02-14 22:54:15,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37536.92 MB 2025-02-14 22:54:15,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 22:54:15,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34619.59 MB 2025-02-14 22:54:15,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:54:15,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:54:15,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 22:54:15,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:15,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24672.52 MB 2025-02-14 22:54:15,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25439.52 MB 2025-02-14 22:54:15,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:54:15,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37536.92 MB 2025-02-14 22:54:15,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37950.06 MB 2025-02-14 22:54:15,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 22:54:15,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26147.31 MB 2025-02-14 22:54:15,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:54:15,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:54:15,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:54:15,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:15,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25852.41 MB 2025-02-14 22:54:15,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26082.38 MB 2025-02-14 22:54:15,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.97 MB 2025-02-14 22:54:15,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37950.06 MB 2025-02-14 22:54:15,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37950.06 MB 2025-02-14 22:54:15,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:54:15,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26245.74 MB 2025-02-14 22:54:15,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:54:15,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:54:15,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.46 seconds 2025-02-14 22:54:15,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:15,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26373.13 MB 2025-02-14 22:54:15,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26283.45 MB 2025-02-14 22:54:15,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -89.68 MB 2025-02-14 22:54:15,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54003.76 MB 2025-02-14 22:54:15,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37950.06 MB 2025-02-14 22:54:15,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16053.70 MB 2025-02-14 22:54:15,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26283.45 MB 2025-02-14 22:54:15,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:54:15,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:54:15,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:54:15,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:15,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26283.45 MB 2025-02-14 22:54:15,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19781.33 MB 2025-02-14 22:54:15,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6502.12 MB 2025-02-14 22:54:15,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37950.06 MB 2025-02-14 22:54:15,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37950.06 MB 2025-02-14 22:54:15,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:54:15,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28795.12 MB 2025-02-14 22:54:15,707 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:54:15,707 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:54:15,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:54:15,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:54:15,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:54:15,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:54:15,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19781.33 MB 2025-02-14 22:54:15,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28220.78 MB 2025-02-14 22:54:15,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.45 MB 2025-02-14 22:54:15,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37950.06 MB 2025-02-14 22:54:15,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46340.77 MB 2025-02-14 22:54:15,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:54:15,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28220.78 MB 2025-02-14 22:54:15,869 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:54:15,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:54:15,871 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:54:15,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:54:15,871 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:54:15,876 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:54:15,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:54:15,877 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:54:15,877 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:55:59,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:55:59,909 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:55:59,914 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:55:59,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:55:59,918 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 165, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:55:59,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:55:59,919 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 165, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:56:02,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:56:02,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:56:02,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.54 seconds 2025-02-14 22:56:02,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:02,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14118.45 MB 2025-02-14 22:56:02,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14702.38 MB 2025-02-14 22:56:02,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 583.93 MB 2025-02-14 22:56:02,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58925.78 MB 2025-02-14 22:56:02,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-14 22:56:02,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39158.02 MB 2025-02-14 22:56:02,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23589.82 MB 2025-02-14 22:56:02,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:56:02,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:56:02,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:56:02,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:02,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.38 MB 2025-02-14 22:56:02,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14859.88 MB 2025-02-14 22:56:02,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 157.50 MB 2025-02-14 22:56:02,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-14 22:56:02,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-14 22:56:02,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:56:02,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16786.70 MB 2025-02-14 22:56:03,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:56:03,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:56:03,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 22:56:03,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14859.88 MB 2025-02-14 22:56:03,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15054.96 MB 2025-02-14 22:56:03,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-14 22:56:03,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-14 22:56:03,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-14 22:56:03,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:56:03,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19029.53 MB 2025-02-14 22:56:03,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:56:03,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:56:03,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:56:03,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15054.90 MB 2025-02-14 22:56:03,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15749.13 MB 2025-02-14 22:56:03,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-14 22:56:03,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-14 22:56:03,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-14 22:56:03,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:56:03,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16270.04 MB 2025-02-14 22:56:03,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:56:03,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:56:03,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:56:03,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15749.13 MB 2025-02-14 22:56:03,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16573.05 MB 2025-02-14 22:56:03,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-14 22:56:03,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-14 22:56:03,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-14 22:56:03,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:56:03,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18610.54 MB 2025-02-14 22:56:03,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:56:03,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:56:03,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:56:03,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15054.90 MB 2025-02-14 22:56:03,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16573.05 MB 2025-02-14 22:56:03,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-14 22:56:03,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-14 22:56:03,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19767.75 MB 2025-02-14 22:56:03,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:56:03,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18610.54 MB 2025-02-14 22:56:03,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:56:03,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:56:03,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 22:56:03,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17136.63 MB 2025-02-14 22:56:03,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17418.50 MB 2025-02-14 22:56:03,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.87 MB 2025-02-14 22:56:03,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19767.75 MB 2025-02-14 22:56:03,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19916.65 MB 2025-02-14 22:56:03,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 22:56:03,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17688.25 MB 2025-02-14 22:56:03,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:56:03,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:56:03,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:56:03,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17570.25 MB 2025-02-14 22:56:03,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17779.43 MB 2025-02-14 22:56:03,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.18 MB 2025-02-14 22:56:03,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19916.65 MB 2025-02-14 22:56:03,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19916.65 MB 2025-02-14 22:56:03,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:56:03,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17789.82 MB 2025-02-14 22:56:03,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:56:03,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:56:03,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.42 seconds 2025-02-14 22:56:03,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13543.58 MB 2025-02-14 22:56:03,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17980.09 MB 2025-02-14 22:56:03,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4436.51 MB 2025-02-14 22:56:03,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58925.78 MB 2025-02-14 22:56:03,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19916.65 MB 2025-02-14 22:56:03,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39009.12 MB 2025-02-14 22:56:03,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17980.09 MB 2025-02-14 22:56:03,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:56:03,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:56:03,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:56:03,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17980.09 MB 2025-02-14 22:56:03,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17347.51 MB 2025-02-14 22:56:03,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -632.57 MB 2025-02-14 22:56:03,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19916.65 MB 2025-02-14 22:56:03,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19916.65 MB 2025-02-14 22:56:03,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:56:03,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19082.92 MB 2025-02-14 22:56:03,628 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 22:56:03,629 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 22:56:03,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:56:03,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:56:03,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:56:03,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:56:03,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17347.51 MB 2025-02-14 22:56:03,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25769.47 MB 2025-02-14 22:56:03,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 22:56:03,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19916.65 MB 2025-02-14 22:56:03,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30381.44 MB 2025-02-14 22:56:03,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 22:56:03,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25769.47 MB 2025-02-14 22:56:03,796 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 22:56:03,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:56:03,797 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:56:03,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:56:03,798 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:56:03,803 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:56:03,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:56:03,804 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:56:03,804 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 22:56:55,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:56:55,065 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:56:55,070 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:56:55,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:56:55,074 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2099, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:56:55,075 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:56:55,075 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2099, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:57:27,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:57:27,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:57:27,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.22 seconds 2025-02-14 22:57:27,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:27,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27594.87 MB 2025-02-14 22:57:27,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35023.12 MB 2025-02-14 22:57:27,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7428.24 MB 2025-02-14 22:57:27,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38753.27 MB 2025-02-14 22:57:27,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38103.15 MB 2025-02-14 22:57:27,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -650.12 MB 2025-02-14 22:57:27,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43861.83 MB 2025-02-14 22:57:27,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:57:27,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:57:27,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 22:57:27,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:27,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35023.12 MB 2025-02-14 22:57:27,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26690.95 MB 2025-02-14 22:57:27,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8332.17 MB 2025-02-14 22:57:27,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38103.15 MB 2025-02-14 22:57:27,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67729.62 MB 2025-02-14 22:57:27,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29626.47 MB 2025-02-14 22:57:27,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55578.93 MB 2025-02-14 22:57:29,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:57:29,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:57:29,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 22:57:29,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:29,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26690.95 MB 2025-02-14 22:57:29,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27221.79 MB 2025-02-14 22:57:29,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:57:29,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67729.62 MB 2025-02-14 22:57:29,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33506.20 MB 2025-02-14 22:57:29,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34223.42 MB 2025-02-14 22:57:29,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31200.33 MB 2025-02-14 22:57:29,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:57:29,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:57:29,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:57:29,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:29,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27221.79 MB 2025-02-14 22:57:29,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29111.32 MB 2025-02-14 22:57:29,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:57:29,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33506.20 MB 2025-02-14 22:57:29,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33506.20 MB 2025-02-14 22:57:29,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:57:29,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30528.75 MB 2025-02-14 22:57:29,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:57:29,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:57:29,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 22:57:29,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:29,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29111.32 MB 2025-02-14 22:57:29,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31353.18 MB 2025-02-14 22:57:29,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:57:29,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33506.20 MB 2025-02-14 22:57:29,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38696.65 MB 2025-02-14 22:57:29,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 22:57:29,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36897.46 MB 2025-02-14 22:57:29,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:57:29,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:57:29,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:57:29,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:29,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27221.79 MB 2025-02-14 22:57:29,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31353.18 MB 2025-02-14 22:57:29,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:57:29,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33506.20 MB 2025-02-14 22:57:29,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38696.65 MB 2025-02-14 22:57:29,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 22:57:29,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36897.46 MB 2025-02-14 22:57:29,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:57:29,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:57:29,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:57:29,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:29,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32886.72 MB 2025-02-14 22:57:29,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33653.72 MB 2025-02-14 22:57:29,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:57:29,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38696.65 MB 2025-02-14 22:57:29,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-14 22:57:29,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 22:57:29,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34361.51 MB 2025-02-14 22:57:29,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:57:29,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:57:29,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:57:29,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:29,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34066.61 MB 2025-02-14 22:57:29,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34295.82 MB 2025-02-14 22:57:29,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.21 MB 2025-02-14 22:57:29,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39111.88 MB 2025-02-14 22:57:29,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-14 22:57:29,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:57:29,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34521.40 MB 2025-02-14 22:57:29,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:57:29,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:57:29,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.74 seconds 2025-02-14 22:57:29,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:29,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20281.79 MB 2025-02-14 22:57:29,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34496.89 MB 2025-02-14 22:57:29,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14215.10 MB 2025-02-14 22:57:29,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38753.27 MB 2025-02-14 22:57:29,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-14 22:57:29,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 358.61 MB 2025-02-14 22:57:29,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34521.40 MB 2025-02-14 22:57:30,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:57:30,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:57:30,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:57:30,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:30,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34496.89 MB 2025-02-14 22:57:30,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25286.18 MB 2025-02-14 22:57:30,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9210.71 MB 2025-02-14 22:57:30,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39111.88 MB 2025-02-14 22:57:30,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39111.88 MB 2025-02-14 22:57:30,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:57:30,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37008.56 MB 2025-02-14 22:57:30,106 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 22:57:30,106 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:57:30,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:57:30,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:57:30,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:57:30,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:30,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25286.18 MB 2025-02-14 22:57:30,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33725.20 MB 2025-02-14 22:57:30,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 22:57:30,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39111.88 MB 2025-02-14 22:57:30,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47502.59 MB 2025-02-14 22:57:30,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 22:57:30,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33725.20 MB 2025-02-14 22:57:30,269 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 22:57:30,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:57:30,270 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:57:30,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:57:30,271 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:57:30,276 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:57:30,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:57:30,277 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:57:30,277 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:57:39,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:57:39,932 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:57:39,937 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:57:39,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:57:39,940 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1126, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:57:39,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:57:39,941 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1126, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:57:57,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:57:57,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:57:57,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.54 seconds 2025-02-14 22:57:57,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:57,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20814.86 MB 2025-02-14 22:57:57,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24799.71 MB 2025-02-14 22:57:57,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3984.85 MB 2025-02-14 22:57:57,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60087.60 MB 2025-02-14 22:57:57,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26289.90 MB 2025-02-14 22:57:57,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33797.70 MB 2025-02-14 22:57:57,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33684.42 MB 2025-02-14 22:57:57,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:57:57,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:57:57,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 22:57:57,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:57,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24799.71 MB 2025-02-14 22:57:57,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21632.62 MB 2025-02-14 22:57:57,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3167.08 MB 2025-02-14 22:57:57,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26289.90 MB 2025-02-14 22:57:57,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44631.59 MB 2025-02-14 22:57:57,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18341.69 MB 2025-02-14 22:57:57,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36338.57 MB 2025-02-14 22:57:59,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:57:59,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:57:59,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 22:57:59,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:59,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21632.62 MB 2025-02-14 22:57:59,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22163.46 MB 2025-02-14 22:57:59,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 22:57:59,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44631.59 MB 2025-02-14 22:57:59,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25136.46 MB 2025-02-14 22:57:59,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19495.12 MB 2025-02-14 22:57:59,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26143.05 MB 2025-02-14 22:57:59,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:57:59,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:57:59,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:57:59,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:59,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22163.46 MB 2025-02-14 22:57:59,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24053.00 MB 2025-02-14 22:57:59,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 22:57:59,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25136.46 MB 2025-02-14 22:57:59,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27023.90 MB 2025-02-14 22:57:59,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 22:57:59,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25470.43 MB 2025-02-14 22:57:59,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:57:59,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:57:59,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 22:57:59,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:59,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24053.00 MB 2025-02-14 22:57:59,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26294.85 MB 2025-02-14 22:57:59,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 22:57:59,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27023.90 MB 2025-02-14 22:57:59,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33629.93 MB 2025-02-14 22:57:59,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 22:57:59,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.14 MB 2025-02-14 22:57:59,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:57:59,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:57:59,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 22:57:59,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:59,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22163.46 MB 2025-02-14 22:57:59,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26294.85 MB 2025-02-14 22:57:59,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 22:57:59,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25136.46 MB 2025-02-14 22:57:59,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33629.93 MB 2025-02-14 22:57:59,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 22:57:59,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.14 MB 2025-02-14 22:57:59,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:57:59,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:57:59,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 22:57:59,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:59,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27828.40 MB 2025-02-14 22:57:59,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28595.40 MB 2025-02-14 22:57:59,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 22:57:59,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33629.93 MB 2025-02-14 22:57:59,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-14 22:57:59,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 22:57:59,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29303.19 MB 2025-02-14 22:57:59,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:57:59,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:57:59,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:57:59,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:59,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29008.29 MB 2025-02-14 22:57:59,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29236.63 MB 2025-02-14 22:57:59,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 22:57:59,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34043.07 MB 2025-02-14 22:57:59,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-14 22:57:59,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:57:59,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29432.13 MB 2025-02-14 22:57:59,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:57:59,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:57:59,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.99 seconds 2025-02-14 22:57:59,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:57:59,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16891.78 MB 2025-02-14 22:57:59,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29436.90 MB 2025-02-14 22:57:59,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12545.11 MB 2025-02-14 22:57:59,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60087.60 MB 2025-02-14 22:57:59,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-14 22:57:59,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26044.53 MB 2025-02-14 22:57:59,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29436.90 MB 2025-02-14 22:58:00,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:58:00,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:58:00,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 22:58:00,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:58:00,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29436.90 MB 2025-02-14 22:58:00,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21883.82 MB 2025-02-14 22:58:00,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7553.07 MB 2025-02-14 22:58:00,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34043.07 MB 2025-02-14 22:58:00,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34043.07 MB 2025-02-14 22:58:00,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:58:00,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31938.42 MB 2025-02-14 22:58:00,223 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 22:58:00,224 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:58:00,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:58:00,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:58:00,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:58:00,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:58:00,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21883.82 MB 2025-02-14 22:58:00,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30288.93 MB 2025-02-14 22:58:00,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 22:58:00,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34043.07 MB 2025-02-14 22:58:00,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42400.22 MB 2025-02-14 22:58:00,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 22:58:00,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30288.93 MB 2025-02-14 22:58:00,385 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 22:58:00,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:58:00,387 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:58:00,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:58:00,388 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:58:00,392 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:58:00,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:58:00,393 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:58:00,393 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 22:59:10,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:59:10,039 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 22:59:10,044 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 22:59:10,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:59:10,048 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 22:59:10,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:59:10,049 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 22:59:12,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 22:59:12,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 22:59:12,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.49 seconds 2025-02-14 22:59:12,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:12,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 22:59:12,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 22:59:12,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 22:59:12,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54934.90 MB 2025-02-14 22:59:12,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 22:59:12,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34932.26 MB 2025-02-14 22:59:12,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-14 22:59:12,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 22:59:12,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 22:59:12,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:59:12,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:12,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 22:59:12,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-14 22:59:12,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-14 22:59:12,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 22:59:12,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 22:59:12,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:59:12,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.75 MB 2025-02-14 22:59:13,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 22:59:13,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 22:59:13,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 22:59:13,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-14 22:59:13,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-14 22:59:13,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 22:59:13,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 22:59:13,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:59:13,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 22:59:13,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19107.09 MB 2025-02-14 22:59:13,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 22:59:13,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 22:59:13,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 22:59:13,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 22:59:13,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-14 22:59:13,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 22:59:13,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:59:13,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 22:59:13,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:59:13,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-14 22:59:13,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 22:59:13,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 22:59:13,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:59:13,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-14 22:59:13,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 22:59:13,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 22:59:13,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:59:13,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20294.14 MB 2025-02-14 22:59:13,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 763.36 MB 2025-02-14 22:59:13,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 22:59:13,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 22:59:13,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 22:59:13,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 22:59:13,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 22:59:13,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 22:59:13,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 22:59:13,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 22:59:13,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20294.14 MB 2025-02-14 22:59:13,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 763.36 MB 2025-02-14 22:59:13,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 22:59:13,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 22:59:13,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 22:59:13,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 22:59:13,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-14 22:59:13,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.71 MB 2025-02-14 22:59:13,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 22:59:13,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20294.14 MB 2025-02-14 22:59:13,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:59:13,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 22:59:13,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18032.02 MB 2025-02-14 22:59:13,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 22:59:13,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 22:59:13,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 22:59:13,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.90 MB 2025-02-14 22:59:13,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18132.30 MB 2025-02-14 22:59:13,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.39 MB 2025-02-14 22:59:13,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20459.81 MB 2025-02-14 22:59:13,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:59:13,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:59:13,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18142.90 MB 2025-02-14 22:59:13,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 22:59:13,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 22:59:13,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-14 22:59:13,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 22:59:13,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18332.93 MB 2025-02-14 22:59:13,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4803.28 MB 2025-02-14 22:59:13,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54934.90 MB 2025-02-14 22:59:13,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:59:13,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34475.08 MB 2025-02-14 22:59:13,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18332.93 MB 2025-02-14 22:59:13,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 22:59:13,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 22:59:13,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 22:59:13,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18332.93 MB 2025-02-14 22:59:13,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17399.26 MB 2025-02-14 22:59:13,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -933.66 MB 2025-02-14 22:59:13,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20459.81 MB 2025-02-14 22:59:13,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20459.81 MB 2025-02-14 22:59:13,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 22:59:13,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19134.89 MB 2025-02-14 22:59:13,791 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 22:59:13,791 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 22:59:13,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 22:59:13,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 22:59:13,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 22:59:13,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 22:59:13,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17399.26 MB 2025-02-14 22:59:13,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.04 MB 2025-02-14 22:59:13,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 22:59:13,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20459.81 MB 2025-02-14 22:59:13,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30924.60 MB 2025-02-14 22:59:13,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 22:59:13,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25820.04 MB 2025-02-14 22:59:13,956 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 22:59:13,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:59:13,957 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 22:59:13,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:59:13,958 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 22:59:13,963 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 22:59:13,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 22:59:13,964 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 22:59:13,964 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 23:00:40,493 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:00:40,494 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:00:40,499 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:00:40,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:00:40,504 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1631, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:00:40,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:00:40,505 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1631, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:01:05,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:01:05,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:01:05,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.03 seconds 2025-02-14 23:01:05,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:05,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24333.78 MB 2025-02-14 23:01:05,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30105.79 MB 2025-02-14 23:01:05,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5772.02 MB 2025-02-14 23:01:05,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39296.43 MB 2025-02-14 23:01:05,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36444.31 MB 2025-02-14 23:01:05,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2852.13 MB 2025-02-14 23:01:05,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39015.28 MB 2025-02-14 23:01:05,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:01:05,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:01:05,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 23:01:05,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:05,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30105.79 MB 2025-02-14 23:01:05,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.91 MB 2025-02-14 23:01:05,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5848.88 MB 2025-02-14 23:01:05,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36444.31 MB 2025-02-14 23:01:05,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55389.98 MB 2025-02-14 23:01:05,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18945.67 MB 2025-02-14 23:01:05,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46784.87 MB 2025-02-14 23:01:07,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:01:07,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:01:07,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 23:01:07,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:07,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24256.91 MB 2025-02-14 23:01:07,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24787.75 MB 2025-02-14 23:01:07,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:01:07,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55389.98 MB 2025-02-14 23:01:07,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27900.51 MB 2025-02-14 23:01:07,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27489.47 MB 2025-02-14 23:01:07,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28766.30 MB 2025-02-14 23:01:07,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:01:07,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:01:07,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:01:07,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:07,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.75 MB 2025-02-14 23:01:07,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26677.29 MB 2025-02-14 23:01:07,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:01:07,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27900.51 MB 2025-02-14 23:01:07,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30731.67 MB 2025-02-14 23:01:07,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 23:01:07,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28094.72 MB 2025-02-14 23:01:07,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:01:07,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:01:07,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 23:01:07,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:07,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26677.29 MB 2025-02-14 23:01:07,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28919.14 MB 2025-02-14 23:01:07,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:01:07,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30731.67 MB 2025-02-14 23:01:07,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36393.98 MB 2025-02-14 23:01:07,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:01:07,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34463.42 MB 2025-02-14 23:01:07,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:01:07,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:01:07,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 23:01:07,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:07,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.75 MB 2025-02-14 23:01:07,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28919.14 MB 2025-02-14 23:01:07,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:01:07,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27900.51 MB 2025-02-14 23:01:07,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36393.98 MB 2025-02-14 23:01:07,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 23:01:07,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34463.42 MB 2025-02-14 23:01:07,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:01:07,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:01:07,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:01:07,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:07,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30452.69 MB 2025-02-14 23:01:07,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31219.69 MB 2025-02-14 23:01:07,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:01:07,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36393.98 MB 2025-02-14 23:01:07,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36811.31 MB 2025-02-14 23:01:07,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:01:07,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31927.48 MB 2025-02-14 23:01:08,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:01:08,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:01:08,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:01:08,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:08,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31632.58 MB 2025-02-14 23:01:08,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31861.97 MB 2025-02-14 23:01:08,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.39 MB 2025-02-14 23:01:08,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36811.31 MB 2025-02-14 23:01:08,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36811.31 MB 2025-02-14 23:01:08,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:01:08,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32072.65 MB 2025-02-14 23:01:08,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:01:08,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:01:08,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.51 seconds 2025-02-14 23:01:08,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:08,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18651.24 MB 2025-02-14 23:01:08,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32062.74 MB 2025-02-14 23:01:08,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13411.50 MB 2025-02-14 23:01:08,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39296.43 MB 2025-02-14 23:01:08,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36811.31 MB 2025-02-14 23:01:08,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2485.13 MB 2025-02-14 23:01:08,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32072.65 MB 2025-02-14 23:01:08,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:01:08,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:01:08,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:01:08,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:08,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32062.74 MB 2025-02-14 23:01:08,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23651.06 MB 2025-02-14 23:01:08,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8411.69 MB 2025-02-14 23:01:08,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36811.31 MB 2025-02-14 23:01:08,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36811.31 MB 2025-02-14 23:01:08,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:01:08,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34570.73 MB 2025-02-14 23:01:08,303 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 23:01:08,303 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:01:08,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:01:08,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:01:08,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:01:08,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:01:08,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23651.06 MB 2025-02-14 23:01:08,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32077.56 MB 2025-02-14 23:01:08,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 23:01:08,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36811.31 MB 2025-02-14 23:01:08,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45189.43 MB 2025-02-14 23:01:08,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 23:01:08,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32077.56 MB 2025-02-14 23:01:08,545 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 23:01:08,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:01:08,547 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:01:08,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:01:08,549 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:01:08,556 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:01:08,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:01:08,558 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:01:08,559 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:01:31,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:01:31,871 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:01:31,876 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:01:31,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:01:31,880 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1853, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:01:31,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:01:31,881 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1853, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:02:00,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:02:00,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:02:00,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.68 seconds 2025-02-14 23:02:00,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:00,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25880.71 MB 2025-02-14 23:02:00,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32438.50 MB 2025-02-14 23:02:00,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6557.79 MB 2025-02-14 23:02:00,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57755.57 MB 2025-02-14 23:02:00,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37230.74 MB 2025-02-14 23:02:00,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20524.83 MB 2025-02-14 23:02:00,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41241.69 MB 2025-02-14 23:02:00,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:02:00,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:02:00,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 23:02:00,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:00,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32438.50 MB 2025-02-14 23:02:00,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25411.02 MB 2025-02-14 23:02:00,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7027.48 MB 2025-02-14 23:02:00,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37230.74 MB 2025-02-14 23:02:00,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60930.65 MB 2025-02-14 23:02:00,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23699.91 MB 2025-02-14 23:02:00,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51455.30 MB 2025-02-14 23:02:02,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:02:02,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:02:02,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 23:02:02,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:02,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25411.02 MB 2025-02-14 23:02:02,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25941.86 MB 2025-02-14 23:02:02,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:02:02,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60930.65 MB 2025-02-14 23:02:02,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-14 23:02:02,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28842.13 MB 2025-02-14 23:02:02,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29920.41 MB 2025-02-14 23:02:02,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:02:02,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:02:02,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:02:02,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:02,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25941.86 MB 2025-02-14 23:02:02,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27831.40 MB 2025-02-14 23:02:02,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:02:02,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 23:02:02,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32088.52 MB 2025-02-14 23:02:02,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:02:02,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29248.82 MB 2025-02-14 23:02:02,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:02:02,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:02:02,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:02:02,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:02,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27831.40 MB 2025-02-14 23:02:02,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30073.25 MB 2025-02-14 23:02:02,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:02:02,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 23:02:02,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37750.83 MB 2025-02-14 23:02:02,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:02:02,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35617.53 MB 2025-02-14 23:02:02,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:02:02,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:02:02,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:02:02,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:02,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25941.86 MB 2025-02-14 23:02:02,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30073.25 MB 2025-02-14 23:02:02,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:02:02,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32088.52 MB 2025-02-14 23:02:02,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37750.83 MB 2025-02-14 23:02:02,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:02:02,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35617.53 MB 2025-02-14 23:02:03,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:02:03,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:02:03,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:02:03,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:03,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31606.79 MB 2025-02-14 23:02:03,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32373.80 MB 2025-02-14 23:02:03,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:02:03,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37750.83 MB 2025-02-14 23:02:03,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38168.17 MB 2025-02-14 23:02:03,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:02:03,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33081.58 MB 2025-02-14 23:02:03,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:02:03,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:02:03,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:02:03,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:03,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32786.68 MB 2025-02-14 23:02:03,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33015.60 MB 2025-02-14 23:02:03,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 23:02:03,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38168.17 MB 2025-02-14 23:02:03,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38168.17 MB 2025-02-14 23:02:03,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:02:03,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33232.71 MB 2025-02-14 23:02:03,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:02:03,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:02:03,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.16 seconds 2025-02-14 23:02:03,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:03,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19424.71 MB 2025-02-14 23:02:03,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33216.42 MB 2025-02-14 23:02:03,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13791.72 MB 2025-02-14 23:02:03,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57755.57 MB 2025-02-14 23:02:03,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38168.17 MB 2025-02-14 23:02:03,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19587.40 MB 2025-02-14 23:02:03,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33232.71 MB 2025-02-14 23:02:03,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:02:03,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:02:03,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:02:03,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:03,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33216.42 MB 2025-02-14 23:02:03,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24425.29 MB 2025-02-14 23:02:03,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8791.14 MB 2025-02-14 23:02:03,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38168.17 MB 2025-02-14 23:02:03,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38168.17 MB 2025-02-14 23:02:03,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:02:03,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35725.02 MB 2025-02-14 23:02:03,331 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 23:02:03,331 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:02:03,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:02:03,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:02:03,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:02:03,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:02:03,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24425.29 MB 2025-02-14 23:02:03,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32854.41 MB 2025-02-14 23:02:03,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 23:02:03,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38168.17 MB 2025-02-14 23:02:03,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46548.39 MB 2025-02-14 23:02:03,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 23:02:03,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32854.41 MB 2025-02-14 23:02:03,493 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 23:02:03,494 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:02:03,494 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:02:03,495 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:02:03,495 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:02:03,500 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:02:03,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:02:03,501 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:02:03,501 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:03:48,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:03:48,077 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:03:48,082 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:03:48,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:03:48,087 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 442, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:03:48,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:03:48,089 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 442, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:03:54,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:03:54,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:03:54,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.79 seconds 2025-02-14 23:03:54,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:54,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.63 MB 2025-02-14 23:03:54,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17612.85 MB 2025-02-14 23:03:54,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1564.21 MB 2025-02-14 23:03:54,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54928.61 MB 2025-02-14 23:03:54,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20946.35 MB 2025-02-14 23:03:54,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33982.25 MB 2025-02-14 23:03:54,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26425.97 MB 2025-02-14 23:03:54,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:03:54,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:03:54,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 23:03:54,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:54,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17612.85 MB 2025-02-14 23:03:54,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18020.53 MB 2025-02-14 23:03:54,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 407.69 MB 2025-02-14 23:03:54,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20946.35 MB 2025-02-14 23:03:54,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25090.33 MB 2025-02-14 23:03:54,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4143.97 MB 2025-02-14 23:03:54,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23156.91 MB 2025-02-14 23:03:56,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:03:56,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:03:56,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 23:03:56,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:56,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18020.53 MB 2025-02-14 23:03:56,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18540.76 MB 2025-02-14 23:03:56,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 520.22 MB 2025-02-14 23:03:56,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25090.33 MB 2025-02-14 23:03:56,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 23:03:56,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3699.38 MB 2025-02-14 23:03:56,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22530.96 MB 2025-02-14 23:03:56,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:03:56,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:03:56,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:03:56,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:56,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18540.76 MB 2025-02-14 23:03:56,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20392.54 MB 2025-02-14 23:03:56,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1851.79 MB 2025-02-14 23:03:56,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 23:03:56,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23240.64 MB 2025-02-14 23:03:56,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1849.69 MB 2025-02-14 23:03:56,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21781.63 MB 2025-02-14 23:03:57,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:03:57,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:03:57,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:03:57,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:57,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20392.54 MB 2025-02-14 23:03:57,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.56 MB 2025-02-14 23:03:57,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2197.02 MB 2025-02-14 23:03:57,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23240.64 MB 2025-02-14 23:03:57,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30412.90 MB 2025-02-14 23:03:57,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7172.26 MB 2025-02-14 23:03:57,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28026.11 MB 2025-02-14 23:03:57,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:03:57,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:03:57,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:03:57,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:57,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18540.76 MB 2025-02-14 23:03:57,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.56 MB 2025-02-14 23:03:57,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4048.81 MB 2025-02-14 23:03:57,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 23:03:57,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30412.90 MB 2025-02-14 23:03:57,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9021.95 MB 2025-02-14 23:03:57,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28026.11 MB 2025-02-14 23:03:57,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:03:57,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:03:57,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:03:57,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:57,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24092.44 MB 2025-02-14 23:03:57,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24846.20 MB 2025-02-14 23:03:57,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 753.76 MB 2025-02-14 23:03:57,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30412.90 MB 2025-02-14 23:03:57,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30819.75 MB 2025-02-14 23:03:57,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 406.85 MB 2025-02-14 23:03:57,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25539.83 MB 2025-02-14 23:03:57,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:03:57,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:03:57,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:03:57,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:57,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.83 MB 2025-02-14 23:03:57,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25482.22 MB 2025-02-14 23:03:57,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.39 MB 2025-02-14 23:03:57,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30819.75 MB 2025-02-14 23:03:57,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30819.75 MB 2025-02-14 23:03:57,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:03:57,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25639.40 MB 2025-02-14 23:03:57,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:03:57,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:03:57,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.14 seconds 2025-02-14 23:03:57,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:57,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14508.67 MB 2025-02-14 23:03:57,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25683.29 MB 2025-02-14 23:03:57,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11174.62 MB 2025-02-14 23:03:57,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54928.61 MB 2025-02-14 23:03:57,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30819.75 MB 2025-02-14 23:03:57,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24108.86 MB 2025-02-14 23:03:57,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25683.29 MB 2025-02-14 23:03:57,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:03:57,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:03:57,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:03:57,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:57,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25683.29 MB 2025-02-14 23:03:57,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19477.40 MB 2025-02-14 23:03:57,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6205.89 MB 2025-02-14 23:03:57,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30819.75 MB 2025-02-14 23:03:57,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30819.75 MB 2025-02-14 23:03:57,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:03:57,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28295.43 MB 2025-02-14 23:03:57,518 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:03:57,518 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:03:57,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:03:57,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:03:57,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:03:57,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:03:57,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19477.40 MB 2025-02-14 23:03:57,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27916.43 MB 2025-02-14 23:03:57,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:03:57,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30819.75 MB 2025-02-14 23:03:57,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39210.45 MB 2025-02-14 23:03:57,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:03:57,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27916.43 MB 2025-02-14 23:03:57,687 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:03:57,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:03:57,689 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:03:57,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:03:57,690 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:03:57,695 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:03:57,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:03:57,696 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:03:57,696 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:04:58,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:04:58,440 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:04:58,445 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:04:58,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:04:58,449 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2470, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:04:58,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:04:58,450 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2470, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:05:36,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:05:36,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:05:36,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.91 seconds 2025-02-14 23:05:36,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:36,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30180.06 MB 2025-02-14 23:05:36,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38921.25 MB 2025-02-14 23:05:36,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8741.19 MB 2025-02-14 23:05:36,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69013.08 MB 2025-02-14 23:05:36,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44075.84 MB 2025-02-14 23:05:36,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24937.23 MB 2025-02-14 23:05:36,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47805.16 MB 2025-02-14 23:05:36,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:05:36,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:05:36,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:05:36,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:36,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38921.25 MB 2025-02-14 23:05:36,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28618.61 MB 2025-02-14 23:05:36,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10302.64 MB 2025-02-14 23:05:36,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44075.84 MB 2025-02-14 23:05:36,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 74708.94 MB 2025-02-14 23:05:36,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30633.10 MB 2025-02-14 23:05:36,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64419.66 MB 2025-02-14 23:05:38,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:05:38,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:05:38,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 23:05:38,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:38,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28618.61 MB 2025-02-14 23:05:38,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29149.45 MB 2025-02-14 23:05:38,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:05:38,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74708.94 MB 2025-02-14 23:05:38,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32375.83 MB 2025-02-14 23:05:38,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42333.11 MB 2025-02-14 23:05:38,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33129.04 MB 2025-02-14 23:05:38,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:05:38,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:05:38,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:05:38,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:38,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29149.45 MB 2025-02-14 23:05:38,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31038.86 MB 2025-02-14 23:05:38,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.40 MB 2025-02-14 23:05:38,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32375.83 MB 2025-02-14 23:05:38,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34263.27 MB 2025-02-14 23:05:38,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 23:05:38,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32456.28 MB 2025-02-14 23:05:38,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:05:38,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:05:38,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:05:38,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:38,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31038.86 MB 2025-02-14 23:05:38,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33280.71 MB 2025-02-14 23:05:38,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:05:38,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34263.27 MB 2025-02-14 23:05:38,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40397.44 MB 2025-02-14 23:05:38,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:05:38,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38824.99 MB 2025-02-14 23:05:38,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:05:38,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:05:38,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:05:38,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:38,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29149.45 MB 2025-02-14 23:05:38,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33280.71 MB 2025-02-14 23:05:38,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.26 MB 2025-02-14 23:05:38,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32375.83 MB 2025-02-14 23:05:38,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40397.44 MB 2025-02-14 23:05:38,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 23:05:38,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38824.99 MB 2025-02-14 23:05:38,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:05:38,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:05:38,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:05:38,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:38,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34814.25 MB 2025-02-14 23:05:38,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35581.26 MB 2025-02-14 23:05:38,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:05:38,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40397.44 MB 2025-02-14 23:05:38,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40812.68 MB 2025-02-14 23:05:38,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:05:38,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.04 MB 2025-02-14 23:05:38,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:05:38,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:05:38,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:05:38,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:38,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35994.14 MB 2025-02-14 23:05:38,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36221.59 MB 2025-02-14 23:05:38,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.45 MB 2025-02-14 23:05:38,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40812.68 MB 2025-02-14 23:05:38,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40812.68 MB 2025-02-14 23:05:38,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:05:38,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36416.37 MB 2025-02-14 23:05:38,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:05:38,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:05:38,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.46 seconds 2025-02-14 23:05:38,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:38,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21574.38 MB 2025-02-14 23:05:38,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36421.51 MB 2025-02-14 23:05:38,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14847.13 MB 2025-02-14 23:05:38,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60404.27 MB 2025-02-14 23:05:38,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40812.68 MB 2025-02-14 23:05:38,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19591.59 MB 2025-02-14 23:05:38,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36421.51 MB 2025-02-14 23:05:39,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:05:39,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:05:39,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:05:39,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:39,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36421.51 MB 2025-02-14 23:05:39,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26561.44 MB 2025-02-14 23:05:39,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9860.08 MB 2025-02-14 23:05:39,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40812.68 MB 2025-02-14 23:05:39,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40812.68 MB 2025-02-14 23:05:39,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:05:39,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38919.31 MB 2025-02-14 23:05:39,203 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-14 23:05:39,204 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:05:39,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:05:39,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:05:39,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:05:39,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:05:39,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26561.44 MB 2025-02-14 23:05:39,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34951.61 MB 2025-02-14 23:05:39,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8390.18 MB 2025-02-14 23:05:39,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40812.68 MB 2025-02-14 23:05:39,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44983.91 MB 2025-02-14 23:05:39,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 23:05:39,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34951.61 MB 2025-02-14 23:05:39,368 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-14 23:05:39,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:05:39,370 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:05:39,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:05:39,371 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:05:39,375 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:05:39,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:05:39,376 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:05:39,376 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:06:22,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:06:22,927 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:06:22,932 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:06:22,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:06:22,936 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1498, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:06:22,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:06:22,937 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1498, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:06:46,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:06:46,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:06:46,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.20 seconds 2025-02-14 23:06:46,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:46,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23407.01 MB 2025-02-14 23:06:46,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28708.61 MB 2025-02-14 23:06:46,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5301.60 MB 2025-02-14 23:06:46,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53326.38 MB 2025-02-14 23:06:46,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35928.41 MB 2025-02-14 23:06:46,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17397.97 MB 2025-02-14 23:06:46,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37635.53 MB 2025-02-14 23:06:46,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:06:46,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:06:46,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 23:06:46,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:46,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28708.61 MB 2025-02-14 23:06:46,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23565.49 MB 2025-02-14 23:06:46,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5143.12 MB 2025-02-14 23:06:46,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35928.41 MB 2025-02-14 23:06:46,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47959.77 MB 2025-02-14 23:06:46,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12031.36 MB 2025-02-14 23:06:46,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43069.69 MB 2025-02-14 23:06:48,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:06:48,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:06:48,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:06:48,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23565.49 MB 2025-02-14 23:06:48,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24096.33 MB 2025-02-14 23:06:48,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:06:48,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47959.77 MB 2025-02-14 23:06:48,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30626.81 MB 2025-02-14 23:06:48,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17332.96 MB 2025-02-14 23:06:48,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28074.87 MB 2025-02-14 23:06:48,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:06:48,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:06:48,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:06:48,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24096.33 MB 2025-02-14 23:06:48,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25985.86 MB 2025-02-14 23:06:48,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:06:48,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30626.81 MB 2025-02-14 23:06:48,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30626.81 MB 2025-02-14 23:06:48,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:06:48,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27403.29 MB 2025-02-14 23:06:48,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:06:48,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:06:48,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:06:48,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25985.86 MB 2025-02-14 23:06:48,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28227.72 MB 2025-02-14 23:06:48,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:06:48,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30626.81 MB 2025-02-14 23:06:48,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36289.12 MB 2025-02-14 23:06:48,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:06:48,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.00 MB 2025-02-14 23:06:48,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:06:48,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:06:48,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 23:06:48,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24096.33 MB 2025-02-14 23:06:48,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28227.72 MB 2025-02-14 23:06:48,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:06:48,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30626.81 MB 2025-02-14 23:06:48,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36289.12 MB 2025-02-14 23:06:48,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:06:48,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.00 MB 2025-02-14 23:06:48,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:06:48,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:06:48,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:06:48,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29761.26 MB 2025-02-14 23:06:48,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30528.26 MB 2025-02-14 23:06:48,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:06:48,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36289.12 MB 2025-02-14 23:06:48,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36706.45 MB 2025-02-14 23:06:48,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:06:48,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31236.05 MB 2025-02-14 23:06:48,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:06:48,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:06:48,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:06:48,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30941.15 MB 2025-02-14 23:06:48,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31170.53 MB 2025-02-14 23:06:48,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.38 MB 2025-02-14 23:06:48,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36706.45 MB 2025-02-14 23:06:48,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36706.45 MB 2025-02-14 23:06:48,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:06:48,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31370.62 MB 2025-02-14 23:06:48,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:06:48,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:06:48,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.64 seconds 2025-02-14 23:06:48,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18187.86 MB 2025-02-14 23:06:48,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31371.60 MB 2025-02-14 23:06:48,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13183.74 MB 2025-02-14 23:06:48,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53326.38 MB 2025-02-14 23:06:48,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36706.45 MB 2025-02-14 23:06:48,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16619.93 MB 2025-02-14 23:06:48,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31371.60 MB 2025-02-14 23:06:48,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:06:48,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:06:48,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:06:48,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31371.60 MB 2025-02-14 23:06:48,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23192.25 MB 2025-02-14 23:06:48,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8179.35 MB 2025-02-14 23:06:48,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36706.45 MB 2025-02-14 23:06:48,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36706.45 MB 2025-02-14 23:06:48,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:06:48,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33883.27 MB 2025-02-14 23:06:48,863 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:06:48,863 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:06:48,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:06:48,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:06:48,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:06:48,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:06:48,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23192.25 MB 2025-02-14 23:06:48,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31631.27 MB 2025-02-14 23:06:48,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:06:48,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36706.45 MB 2025-02-14 23:06:48,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45097.16 MB 2025-02-14 23:06:48,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:06:48,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31631.27 MB 2025-02-14 23:06:49,033 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:06:49,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:06:49,035 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:06:49,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:06:49,036 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:06:49,041 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:06:49,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:06:49,042 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:06:49,042 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:07:05,459 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:07:05,459 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:07:05,464 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:07:05,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:07:05,467 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 806, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:07:05,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:07:05,468 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 806, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:07:18,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:07:18,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:07:18,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.53 seconds 2025-02-14 23:07:18,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:18,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18585.04 MB 2025-02-14 23:07:18,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21437.43 MB 2025-02-14 23:07:18,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2852.39 MB 2025-02-14 23:07:18,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57682.17 MB 2025-02-14 23:07:18,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26457.67 MB 2025-02-14 23:07:18,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31224.50 MB 2025-02-14 23:07:18,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30321.34 MB 2025-02-14 23:07:18,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:07:18,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:07:18,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 23:07:18,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:18,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21437.43 MB 2025-02-14 23:07:18,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19967.99 MB 2025-02-14 23:07:18,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1469.44 MB 2025-02-14 23:07:18,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26457.67 MB 2025-02-14 23:07:18,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35662.07 MB 2025-02-14 23:07:18,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9204.40 MB 2025-02-14 23:07:18,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31247.72 MB 2025-02-14 23:07:19,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:07:19,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:07:19,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:07:19,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:19,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19967.99 MB 2025-02-14 23:07:19,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20498.84 MB 2025-02-14 23:07:19,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:07:19,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35662.07 MB 2025-02-14 23:07:19,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27873.25 MB 2025-02-14 23:07:19,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7788.82 MB 2025-02-14 23:07:19,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24477.38 MB 2025-02-14 23:07:19,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:07:19,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:07:19,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:07:19,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:19,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20498.84 MB 2025-02-14 23:07:19,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22388.37 MB 2025-02-14 23:07:19,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:07:19,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 23:07:19,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27873.25 MB 2025-02-14 23:07:19,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:07:19,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23805.80 MB 2025-02-14 23:07:20,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:07:20,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:07:20,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:07:20,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:20,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22388.37 MB 2025-02-14 23:07:20,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24630.23 MB 2025-02-14 23:07:20,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:07:20,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 23:07:20,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32119.98 MB 2025-02-14 23:07:20,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 23:07:20,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30174.51 MB 2025-02-14 23:07:20,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:07:20,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:07:20,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 23:07:20,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:20,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20498.84 MB 2025-02-14 23:07:20,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24630.23 MB 2025-02-14 23:07:20,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:07:20,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 23:07:20,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32119.98 MB 2025-02-14 23:07:20,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 23:07:20,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30174.51 MB 2025-02-14 23:07:20,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:07:20,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:07:20,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:07:20,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:20,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26163.77 MB 2025-02-14 23:07:20,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26930.77 MB 2025-02-14 23:07:20,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:07:20,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32119.98 MB 2025-02-14 23:07:20,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32535.22 MB 2025-02-14 23:07:20,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:07:20,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27638.56 MB 2025-02-14 23:07:20,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:07:20,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:07:20,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:07:20,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:20,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27343.66 MB 2025-02-14 23:07:20,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27571.93 MB 2025-02-14 23:07:20,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.27 MB 2025-02-14 23:07:20,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32535.22 MB 2025-02-14 23:07:20,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32535.22 MB 2025-02-14 23:07:20,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:07:20,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27782.72 MB 2025-02-14 23:07:20,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:07:20,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:07:20,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.94 seconds 2025-02-14 23:07:20,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:20,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15776.88 MB 2025-02-14 23:07:20,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27772.49 MB 2025-02-14 23:07:20,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11995.61 MB 2025-02-14 23:07:20,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57682.17 MB 2025-02-14 23:07:20,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32535.22 MB 2025-02-14 23:07:20,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25146.95 MB 2025-02-14 23:07:20,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27782.72 MB 2025-02-14 23:07:20,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:07:20,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:07:20,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:07:20,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:20,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27772.49 MB 2025-02-14 23:07:20,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20773.26 MB 2025-02-14 23:07:20,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6999.22 MB 2025-02-14 23:07:20,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32535.22 MB 2025-02-14 23:07:20,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32535.22 MB 2025-02-14 23:07:20,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:07:20,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30277.70 MB 2025-02-14 23:07:20,695 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 23:07:20,696 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:07:20,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:07:20,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:07:20,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 23:07:20,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:07:20,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20773.26 MB 2025-02-14 23:07:20,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29191.01 MB 2025-02-14 23:07:20,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-14 23:07:20,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32535.22 MB 2025-02-14 23:07:20,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40902.85 MB 2025-02-14 23:07:20,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 23:07:20,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29191.01 MB 2025-02-14 23:07:20,866 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 23:07:20,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:07:20,867 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:07:20,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:07:20,868 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:07:20,873 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:07:20,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:07:20,874 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:07:20,874 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:08:45,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:08:45,114 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:08:45,120 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:08:45,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:08:45,123 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 305, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:08:45,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:08:45,124 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 305, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:08:49,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:08:49,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:08:49,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.68 seconds 2025-02-14 23:08:49,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:49,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15094.00 MB 2025-02-14 23:08:49,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16173.37 MB 2025-02-14 23:08:49,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1079.38 MB 2025-02-14 23:08:49,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49270.49 MB 2025-02-14 23:08:49,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21887.98 MB 2025-02-14 23:08:49,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27382.51 MB 2025-02-14 23:08:49,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25018.35 MB 2025-02-14 23:08:49,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:08:49,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:08:49,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:08:49,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:49,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16173.37 MB 2025-02-14 23:08:49,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16612.92 MB 2025-02-14 23:08:49,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.55 MB 2025-02-14 23:08:49,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21887.98 MB 2025-02-14 23:08:49,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23429.38 MB 2025-02-14 23:08:49,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1541.41 MB 2025-02-14 23:08:49,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20294.13 MB 2025-02-14 23:08:51,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:08:51,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:08:51,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.39 seconds 2025-02-14 23:08:51,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16612.92 MB 2025-02-14 23:08:51,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17001.76 MB 2025-02-14 23:08:51,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 388.84 MB 2025-02-14 23:08:51,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23429.38 MB 2025-02-14 23:08:51,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22485.66 MB 2025-02-14 23:08:51,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 23:08:51,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20952.44 MB 2025-02-14 23:08:51,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:08:51,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:08:51,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:08:51,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17001.76 MB 2025-02-14 23:08:51,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18386.41 MB 2025-02-14 23:08:51,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1384.64 MB 2025-02-14 23:08:51,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22485.66 MB 2025-02-14 23:08:51,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22485.66 MB 2025-02-14 23:08:51,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:08:51,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19424.68 MB 2025-02-14 23:08:51,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:08:51,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:08:51,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:08:51,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18386.41 MB 2025-02-14 23:08:51,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20028.59 MB 2025-02-14 23:08:51,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.18 MB 2025-02-14 23:08:51,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22485.66 MB 2025-02-14 23:08:51,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25945.96 MB 2025-02-14 23:08:51,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3460.30 MB 2025-02-14 23:08:51,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24092.90 MB 2025-02-14 23:08:51,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:08:51,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:08:51,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:08:51,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17001.76 MB 2025-02-14 23:08:51,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20028.59 MB 2025-02-14 23:08:51,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3026.82 MB 2025-02-14 23:08:51,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22485.66 MB 2025-02-14 23:08:51,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25945.96 MB 2025-02-14 23:08:51,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3460.30 MB 2025-02-14 23:08:51,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24092.90 MB 2025-02-14 23:08:51,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:08:51,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:08:51,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 23:08:51,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21151.91 MB 2025-02-14 23:08:51,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21713.74 MB 2025-02-14 23:08:51,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 561.83 MB 2025-02-14 23:08:51,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25945.96 MB 2025-02-14 23:08:51,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26250.05 MB 2025-02-14 23:08:51,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 304.09 MB 2025-02-14 23:08:51,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22232.19 MB 2025-02-14 23:08:51,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:08:51,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:08:51,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:08:51,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22016.18 MB 2025-02-14 23:08:51,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22234.58 MB 2025-02-14 23:08:51,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.40 MB 2025-02-14 23:08:51,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26250.05 MB 2025-02-14 23:08:51,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26252.15 MB 2025-02-14 23:08:51,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 23:08:51,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22321.53 MB 2025-02-14 23:08:51,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:08:51,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:08:51,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.41 seconds 2025-02-14 23:08:51,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14031.35 MB 2025-02-14 23:08:51,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22435.65 MB 2025-02-14 23:08:51,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8404.30 MB 2025-02-14 23:08:51,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49270.49 MB 2025-02-14 23:08:51,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26252.15 MB 2025-02-14 23:08:51,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23018.34 MB 2025-02-14 23:08:51,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22435.65 MB 2025-02-14 23:08:51,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:08:51,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:08:51,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:08:51,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22435.65 MB 2025-02-14 23:08:51,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25449.68 MB 2025-02-14 23:08:51,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 23:08:51,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26252.15 MB 2025-02-14 23:08:51,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27057.46 MB 2025-02-14 23:08:51,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 805.31 MB 2025-02-14 23:08:51,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25751.31 MB 2025-02-14 23:08:51,822 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:08:51,823 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:08:51,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:08:51,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:08:51,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:08:51,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:08:51,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18530.78 MB 2025-02-14 23:08:51,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26969.80 MB 2025-02-14 23:08:51,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:08:51,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27057.46 MB 2025-02-14 23:08:51,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35448.16 MB 2025-02-14 23:08:51,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:08:51,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26969.80 MB 2025-02-14 23:08:51,988 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:08:51,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:08:51,989 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:08:51,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:08:51,990 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:08:51,995 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:08:51,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:08:51,996 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:08:51,996 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:09:01,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:09:01,430 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:09:01,435 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:09:01,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:09:01,438 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1810, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:09:01,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:09:01,439 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1810, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:09:29,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:09:29,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:09:29,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.94 seconds 2025-02-14 23:09:29,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:29,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25581.08 MB 2025-02-14 23:09:29,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31986.57 MB 2025-02-14 23:09:29,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6405.49 MB 2025-02-14 23:09:29,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48033.17 MB 2025-02-14 23:09:29,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37104.91 MB 2025-02-14 23:09:29,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10928.26 MB 2025-02-14 23:09:29,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40942.06 MB 2025-02-14 23:09:29,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:09:29,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:09:29,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 23:09:29,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:29,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31986.57 MB 2025-02-14 23:09:29,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25187.48 MB 2025-02-14 23:09:29,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6799.09 MB 2025-02-14 23:09:29,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37104.91 MB 2025-02-14 23:09:29,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59867.40 MB 2025-02-14 23:09:29,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22762.49 MB 2025-02-14 23:09:29,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50543.67 MB 2025-02-14 23:09:31,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:09:31,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:09:31,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 23:09:31,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:31,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25187.48 MB 2025-02-14 23:09:31,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25718.32 MB 2025-02-14 23:09:31,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:09:31,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59867.40 MB 2025-02-14 23:09:31,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32113.69 MB 2025-02-14 23:09:31,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27753.71 MB 2025-02-14 23:09:31,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29696.87 MB 2025-02-14 23:09:31,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:09:31,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:09:31,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:09:31,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:31,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25718.32 MB 2025-02-14 23:09:31,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27607.85 MB 2025-02-14 23:09:31,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:09:31,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32113.69 MB 2025-02-14 23:09:31,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32113.69 MB 2025-02-14 23:09:31,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:09:31,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29025.28 MB 2025-02-14 23:09:31,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:09:31,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:09:31,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:09:31,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:31,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27607.85 MB 2025-02-14 23:09:31,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29849.71 MB 2025-02-14 23:09:31,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:09:31,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32113.69 MB 2025-02-14 23:09:31,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37776.00 MB 2025-02-14 23:09:31,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:09:31,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35393.99 MB 2025-02-14 23:09:31,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:09:31,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:09:31,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:09:31,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:31,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25718.32 MB 2025-02-14 23:09:31,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29849.71 MB 2025-02-14 23:09:31,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:09:31,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32113.69 MB 2025-02-14 23:09:31,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37776.00 MB 2025-02-14 23:09:31,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:09:31,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35393.99 MB 2025-02-14 23:09:31,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:09:31,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:09:31,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:09:31,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:31,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31383.25 MB 2025-02-14 23:09:31,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32150.25 MB 2025-02-14 23:09:31,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:09:31,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37776.00 MB 2025-02-14 23:09:31,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 23:09:31,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:09:31,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32858.04 MB 2025-02-14 23:09:31,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:09:31,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:09:31,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:09:31,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:31,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32563.14 MB 2025-02-14 23:09:31,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32791.75 MB 2025-02-14 23:09:31,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.60 MB 2025-02-14 23:09:31,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38191.24 MB 2025-02-14 23:09:31,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 23:09:31,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:09:31,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33016.86 MB 2025-02-14 23:09:31,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:09:31,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:09:31,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.40 seconds 2025-02-14 23:09:31,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:31,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19274.89 MB 2025-02-14 23:09:31,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32992.62 MB 2025-02-14 23:09:31,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13717.73 MB 2025-02-14 23:09:31,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48033.17 MB 2025-02-14 23:09:31,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 23:09:31,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9841.93 MB 2025-02-14 23:09:31,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33016.86 MB 2025-02-14 23:09:32,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:09:32,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:09:32,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:09:32,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:32,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32992.62 MB 2025-02-14 23:09:32,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24276.23 MB 2025-02-14 23:09:32,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8716.39 MB 2025-02-14 23:09:32,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38191.24 MB 2025-02-14 23:09:32,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38191.24 MB 2025-02-14 23:09:32,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:09:32,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35501.83 MB 2025-02-14 23:09:32,137 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 23:09:32,137 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:09:32,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:09:32,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:09:32,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 23:09:32,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:09:32,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24276.23 MB 2025-02-14 23:09:32,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32706.91 MB 2025-02-14 23:09:32,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 23:09:32,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38191.24 MB 2025-02-14 23:09:32,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46573.55 MB 2025-02-14 23:09:32,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 23:09:32,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32706.91 MB 2025-02-14 23:09:32,300 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 23:09:32,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:09:32,302 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:09:32,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:09:32,303 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:09:32,307 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:09:32,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:09:32,308 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:09:32,309 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:10:32,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:32,364 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:10:32,369 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:10:32,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:32,373 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 172, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:10:32,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:32,374 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 172, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:10:35,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:10:35,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:10:35,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.67 seconds 2025-02-14 23:10:35,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:35,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14167.23 MB 2025-02-14 23:10:35,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14775.93 MB 2025-02-14 23:10:35,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.70 MB 2025-02-14 23:10:35,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59145.98 MB 2025-02-14 23:10:35,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-14 23:10:35,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39141.24 MB 2025-02-14 23:10:35,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23638.60 MB 2025-02-14 23:10:35,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:10:35,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:10:35,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:10:35,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:35,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14775.93 MB 2025-02-14 23:10:35,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14993.59 MB 2025-02-14 23:10:35,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.66 MB 2025-02-14 23:10:35,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-14 23:10:35,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20004.73 MB 2025-02-14 23:10:35,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:10:35,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17079.87 MB 2025-02-14 23:10:35,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:10:35,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:10:35,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 23:10:35,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:35,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14993.59 MB 2025-02-14 23:10:35,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15207.25 MB 2025-02-14 23:10:35,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 23:10:35,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20004.73 MB 2025-02-14 23:10:35,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 23:10:35,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 23:10:35,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19164.28 MB 2025-02-14 23:10:35,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:10:35,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:10:35,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 23:10:35,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:35,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15207.19 MB 2025-02-14 23:10:35,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15967.54 MB 2025-02-14 23:10:35,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 23:10:35,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 23:10:35,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19532.87 MB 2025-02-14 23:10:35,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:10:35,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16538.06 MB 2025-02-14 23:10:35,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:10:35,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:10:35,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 23:10:35,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:35,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15967.54 MB 2025-02-14 23:10:35,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16869.93 MB 2025-02-14 23:10:35,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 23:10:35,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 23:10:35,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20677.92 MB 2025-02-14 23:10:35,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 23:10:35,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19101.46 MB 2025-02-14 23:10:35,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:10:35,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:10:35,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 23:10:35,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:35,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15207.19 MB 2025-02-14 23:10:35,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16869.93 MB 2025-02-14 23:10:35,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 23:10:35,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19532.87 MB 2025-02-14 23:10:35,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20677.92 MB 2025-02-14 23:10:35,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1145.04 MB 2025-02-14 23:10:35,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19101.46 MB 2025-02-14 23:10:35,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:10:35,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:10:35,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 23:10:35,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:35,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17487.18 MB 2025-02-14 23:10:35,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17795.89 MB 2025-02-14 23:10:35,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 23:10:35,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20677.92 MB 2025-02-14 23:10:35,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 23:10:35,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 23:10:35,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18088.69 MB 2025-02-14 23:10:36,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:10:36,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:10:36,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:10:36,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:36,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17962.09 MB 2025-02-14 23:10:36,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18191.03 MB 2025-02-14 23:10:36,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-14 23:10:36,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 23:10:36,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 23:10:36,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:10:36,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18205.95 MB 2025-02-14 23:10:36,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:10:36,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:10:36,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.63 seconds 2025-02-14 23:10:36,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:36,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-14 23:10:36,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18392.11 MB 2025-02-14 23:10:36,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4824.14 MB 2025-02-14 23:10:36,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59145.98 MB 2025-02-14 23:10:36,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 23:10:36,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38304.48 MB 2025-02-14 23:10:36,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18392.11 MB 2025-02-14 23:10:36,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:10:36,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:10:36,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:10:36,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:36,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18392.11 MB 2025-02-14 23:10:36,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17444.45 MB 2025-02-14 23:10:36,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -947.66 MB 2025-02-14 23:10:36,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 23:10:36,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 23:10:36,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:10:36,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19195.84 MB 2025-02-14 23:10:36,294 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:10:36,294 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 23:10:36,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:10:36,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:10:36,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:10:36,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:10:36,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17444.45 MB 2025-02-14 23:10:36,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25883.47 MB 2025-02-14 23:10:36,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:10:36,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 23:10:36,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29232.20 MB 2025-02-14 23:10:36,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:10:36,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25883.47 MB 2025-02-14 23:10:36,457 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:10:36,458 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:36,458 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:10:36,459 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:36,459 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:10:36,464 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:10:36,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:36,465 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:10:36,465 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 23:10:44,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:44,505 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:10:44,510 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:10:44,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:44,514 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:10:44,515 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:10:44,515 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:11:03,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:11:03,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:11:03,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.46 seconds 2025-02-14 23:11:03,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:03,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21797.37 MB 2025-02-14 23:11:03,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26281.21 MB 2025-02-14 23:11:03,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-14 23:11:03,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41817.21 MB 2025-02-14 23:11:03,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35181.82 MB 2025-02-14 23:11:03,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6635.39 MB 2025-02-14 23:11:03,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.11 MB 2025-02-14 23:11:04,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:11:04,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:11:04,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 23:11:04,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:04,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26281.21 MB 2025-02-14 23:11:04,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22364.59 MB 2025-02-14 23:11:04,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-14 23:11:04,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35181.82 MB 2025-02-14 23:11:04,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44356.86 MB 2025-02-14 23:11:04,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9175.04 MB 2025-02-14 23:11:04,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39653.79 MB 2025-02-14 23:11:05,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:11:05,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:11:05,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:11:05,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:05,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22364.59 MB 2025-02-14 23:11:05,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22895.43 MB 2025-02-14 23:11:05,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:11:05,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44356.86 MB 2025-02-14 23:11:05,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26501.71 MB 2025-02-14 23:11:05,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17855.15 MB 2025-02-14 23:11:05,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26875.02 MB 2025-02-14 23:11:05,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:11:05,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:11:05,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:11:05,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:05,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-14 23:11:05,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24784.96 MB 2025-02-14 23:11:05,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:11:05,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26501.71 MB 2025-02-14 23:11:05,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27445.43 MB 2025-02-14 23:11:05,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 23:11:05,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26202.39 MB 2025-02-14 23:11:06,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:11:06,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:11:06,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 23:11:06,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:06,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24784.96 MB 2025-02-14 23:11:06,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-14 23:11:06,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:11:06,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27445.43 MB 2025-02-14 23:11:06,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34051.46 MB 2025-02-14 23:11:06,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:11:06,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-14 23:11:06,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:11:06,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:11:06,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:11:06,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:06,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-14 23:11:06,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-14 23:11:06,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:11:06,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26501.71 MB 2025-02-14 23:11:06,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34051.46 MB 2025-02-14 23:11:06,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 23:11:06,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-14 23:11:06,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:11:06,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:11:06,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:11:06,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:06,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28560.36 MB 2025-02-14 23:11:06,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29327.36 MB 2025-02-14 23:11:06,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:11:06,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34051.46 MB 2025-02-14 23:11:06,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 23:11:06,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:11:06,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30035.15 MB 2025-02-14 23:11:06,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:11:06,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:11:06,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:11:06,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:06,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29740.25 MB 2025-02-14 23:11:06,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29966.92 MB 2025-02-14 23:11:06,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.67 MB 2025-02-14 23:11:06,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 23:11:06,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 23:11:06,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:06,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30210.21 MB 2025-02-14 23:11:06,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:11:06,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:11:06,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.86 seconds 2025-02-14 23:11:06,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:06,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.04 MB 2025-02-14 23:11:06,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30167.40 MB 2025-02-14 23:11:06,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12784.37 MB 2025-02-14 23:11:06,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41817.21 MB 2025-02-14 23:11:06,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 23:11:06,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7348.42 MB 2025-02-14 23:11:06,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30210.21 MB 2025-02-14 23:11:06,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:11:06,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:11:06,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:11:06,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:06,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30167.40 MB 2025-02-14 23:11:06,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22370.09 MB 2025-02-14 23:11:06,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7797.32 MB 2025-02-14 23:11:06,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 23:11:06,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 23:11:06,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:06,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.63 MB 2025-02-14 23:11:06,660 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-14 23:11:06,660 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:11:06,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:11:06,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:11:06,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:11:06,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:06,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22370.09 MB 2025-02-14 23:11:06,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30761.13 MB 2025-02-14 23:11:06,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-14 23:11:06,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 23:11:06,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38640.03 MB 2025-02-14 23:11:06,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 23:11:06,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30761.13 MB 2025-02-14 23:11:06,821 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-14 23:11:06,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:06,823 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:11:06,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:06,824 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:11:06,828 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:11:06,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:06,829 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:11:06,829 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:11:15,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:15,383 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:11:15,388 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:11:15,391 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:15,391 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:11:15,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:15,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:11:17,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:11:17,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:11:17,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.27 seconds 2025-02-14 23:11:17,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:17,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-14 23:11:17,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14492.63 MB 2025-02-14 23:11:17,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.54 MB 2025-02-14 23:11:17,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46982.50 MB 2025-02-14 23:11:17,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20946.35 MB 2025-02-14 23:11:17,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26036.14 MB 2025-02-14 23:11:17,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23450.46 MB 2025-02-14 23:11:17,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:11:17,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:11:17,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:11:17,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:17,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14492.63 MB 2025-02-14 23:11:17,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14740.86 MB 2025-02-14 23:11:17,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.23 MB 2025-02-14 23:11:17,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20946.35 MB 2025-02-14 23:11:17,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20946.35 MB 2025-02-14 23:11:17,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:17,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16533.29 MB 2025-02-14 23:11:18,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:11:18,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:11:18,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 23:11:18,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14740.86 MB 2025-02-14 23:11:18,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14933.29 MB 2025-02-14 23:11:18,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 23:11:18,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20946.35 MB 2025-02-14 23:11:18,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 23:11:18,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 23:11:18,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18910.51 MB 2025-02-14 23:11:18,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:11:18,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:11:18,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 23:11:18,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-14 23:11:18,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15618.01 MB 2025-02-14 23:11:18,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 23:11:18,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 23:11:18,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 23:11:18,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:18,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16131.83 MB 2025-02-14 23:11:18,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:11:18,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:11:18,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:11:18,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15618.01 MB 2025-02-14 23:11:18,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-14 23:11:18,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 23:11:18,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 23:11:18,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 23:11:18,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:18,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18440.49 MB 2025-02-14 23:11:18,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:11:18,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:11:18,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:11:18,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-14 23:11:18,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-14 23:11:18,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 23:11:18,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 23:11:18,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 23:11:18,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:18,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18440.49 MB 2025-02-14 23:11:18,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:11:18,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:11:18,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 23:11:18,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16986.63 MB 2025-02-14 23:11:18,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17264.67 MB 2025-02-14 23:11:18,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 23:11:18,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 23:11:18,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20151.53 MB 2025-02-14 23:11:18,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 23:11:18,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17531.24 MB 2025-02-14 23:11:18,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:11:18,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:11:18,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:11:18,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17414.35 MB 2025-02-14 23:11:18,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17642.14 MB 2025-02-14 23:11:18,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.79 MB 2025-02-14 23:11:18,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20151.53 MB 2025-02-14 23:11:18,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20151.53 MB 2025-02-14 23:11:18,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:18,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17646.24 MB 2025-02-14 23:11:18,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:11:18,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:11:18,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.14 seconds 2025-02-14 23:11:18,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13473.90 MB 2025-02-14 23:11:18,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17842.99 MB 2025-02-14 23:11:18,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4369.09 MB 2025-02-14 23:11:18,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46982.50 MB 2025-02-14 23:11:18,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20151.53 MB 2025-02-14 23:11:18,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26830.96 MB 2025-02-14 23:11:18,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17842.99 MB 2025-02-14 23:11:18,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:11:18,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:11:18,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:11:18,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17842.99 MB 2025-02-14 23:11:18,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17270.82 MB 2025-02-14 23:11:18,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -572.18 MB 2025-02-14 23:11:18,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20151.53 MB 2025-02-14 23:11:18,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20151.53 MB 2025-02-14 23:11:18,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:18,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18946.91 MB 2025-02-14 23:11:18,824 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 23:11:18,824 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 23:11:18,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:11:18,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:11:18,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:11:18,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:18,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17270.82 MB 2025-02-14 23:11:18,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25701.22 MB 2025-02-14 23:11:18,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 23:11:18,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20151.53 MB 2025-02-14 23:11:18,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30626.81 MB 2025-02-14 23:11:18,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 23:11:18,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25701.22 MB 2025-02-14 23:11:18,985 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 23:11:18,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:18,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:11:18,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:18,988 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:11:18,992 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:11:18,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:18,993 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:11:18,994 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 23:11:44,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:44,451 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:11:44,456 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:11:44,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:44,460 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 140, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:11:44,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:44,461 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 140, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:11:46,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:11:46,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:11:46,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.18 seconds 2025-02-14 23:11:46,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:46,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13944.25 MB 2025-02-14 23:11:46,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14439.70 MB 2025-02-14 23:11:46,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 495.45 MB 2025-02-14 23:11:46,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39007.03 MB 2025-02-14 23:11:46,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20000.54 MB 2025-02-14 23:11:46,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19006.49 MB 2025-02-14 23:11:46,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23415.62 MB 2025-02-14 23:11:46,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:11:46,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:11:46,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:11:46,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:46,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14439.70 MB 2025-02-14 23:11:46,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14679.75 MB 2025-02-14 23:11:46,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.05 MB 2025-02-14 23:11:46,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20000.54 MB 2025-02-14 23:11:46,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20000.54 MB 2025-02-14 23:11:46,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:46,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16448.68 MB 2025-02-14 23:11:47,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:11:47,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:11:47,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.67 seconds 2025-02-14 23:11:47,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14679.75 MB 2025-02-14 23:11:47,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14865.54 MB 2025-02-14 23:11:47,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 185.79 MB 2025-02-14 23:11:47,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20000.54 MB 2025-02-14 23:11:47,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 23:11:47,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 23:11:47,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18849.40 MB 2025-02-14 23:11:47,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:11:47,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:11:47,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 23:11:47,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14865.48 MB 2025-02-14 23:11:47,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15526.65 MB 2025-02-14 23:11:47,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 661.18 MB 2025-02-14 23:11:47,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 23:11:47,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 23:11:47,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:47,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16022.76 MB 2025-02-14 23:11:47,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:11:47,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:11:47,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 23:11:47,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15526.65 MB 2025-02-14 23:11:47,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16311.34 MB 2025-02-14 23:11:47,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 784.69 MB 2025-02-14 23:11:47,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 23:11:47,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 23:11:47,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:47,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18251.80 MB 2025-02-14 23:11:47,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:11:47,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:11:47,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:11:47,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14865.48 MB 2025-02-14 23:11:47,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16311.34 MB 2025-02-14 23:11:47,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.87 MB 2025-02-14 23:11:47,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 23:11:47,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19528.68 MB 2025-02-14 23:11:47,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:47,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18251.80 MB 2025-02-14 23:11:47,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:11:47,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:11:47,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 23:11:47,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16848.08 MB 2025-02-14 23:11:47,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17116.53 MB 2025-02-14 23:11:47,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 268.45 MB 2025-02-14 23:11:47,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19528.68 MB 2025-02-14 23:11:47,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19673.38 MB 2025-02-14 23:11:47,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 144.70 MB 2025-02-14 23:11:47,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17373.90 MB 2025-02-14 23:11:47,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:11:47,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:11:47,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:11:47,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17261.05 MB 2025-02-14 23:11:47,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17471.55 MB 2025-02-14 23:11:47,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.50 MB 2025-02-14 23:11:47,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19673.38 MB 2025-02-14 23:11:47,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19673.38 MB 2025-02-14 23:11:47,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:11:47,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17477.28 MB 2025-02-14 23:11:47,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:11:47,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:11:47,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.02 seconds 2025-02-14 23:11:47,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13456.48 MB 2025-02-14 23:11:47,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17672.38 MB 2025-02-14 23:11:47,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4215.90 MB 2025-02-14 23:11:47,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39007.03 MB 2025-02-14 23:11:47,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19673.38 MB 2025-02-14 23:11:47,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19333.64 MB 2025-02-14 23:11:47,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17672.38 MB 2025-02-14 23:11:47,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:11:47,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:11:47,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:11:47,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17672.38 MB 2025-02-14 23:11:47,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20682.73 MB 2025-02-14 23:11:47,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.35 MB 2025-02-14 23:11:47,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19673.38 MB 2025-02-14 23:11:47,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22223.52 MB 2025-02-14 23:11:47,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2550.14 MB 2025-02-14 23:11:47,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20984.15 MB 2025-02-14 23:11:47,766 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 23:11:47,767 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 23:11:47,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:11:47,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:11:47,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:11:47,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:11:47,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20682.73 MB 2025-02-14 23:11:47,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29111.85 MB 2025-02-14 23:11:47,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 23:11:47,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22223.52 MB 2025-02-14 23:11:47,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32698.79 MB 2025-02-14 23:11:47,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 23:11:47,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29111.85 MB 2025-02-14 23:11:47,928 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 23:11:47,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:47,929 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:11:47,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:47,931 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:11:47,935 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:11:47,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:11:47,936 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:11:47,937 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 23:12:21,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:21,710 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:12:21,715 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:12:21,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:21,719 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 495, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:12:21,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:21,720 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 495, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:12:29,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:12:29,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:12:29,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.58 seconds 2025-02-14 23:12:29,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:29,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16417.95 MB 2025-02-14 23:12:29,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18169.72 MB 2025-02-14 23:12:29,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1751.78 MB 2025-02-14 23:12:29,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41081.11 MB 2025-02-14 23:12:29,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21153.97 MB 2025-02-14 23:12:29,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19927.14 MB 2025-02-14 23:12:29,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27021.78 MB 2025-02-14 23:12:29,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:12:29,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:12:29,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 23:12:29,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:29,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18169.72 MB 2025-02-14 23:12:29,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18352.25 MB 2025-02-14 23:12:29,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 182.53 MB 2025-02-14 23:12:29,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21153.97 MB 2025-02-14 23:12:29,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29399.97 MB 2025-02-14 23:12:29,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8246.00 MB 2025-02-14 23:12:29,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25806.84 MB 2025-02-14 23:12:31,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:12:31,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:12:31,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:12:31,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18352.25 MB 2025-02-14 23:12:31,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18883.09 MB 2025-02-14 23:12:31,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:12:31,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29399.97 MB 2025-02-14 23:12:31,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22334.67 MB 2025-02-14 23:12:31,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7065.31 MB 2025-02-14 23:12:31,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22862.68 MB 2025-02-14 23:12:31,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:12:31,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:12:31,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:12:31,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18883.09 MB 2025-02-14 23:12:31,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20772.63 MB 2025-02-14 23:12:31,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:12:31,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22334.67 MB 2025-02-14 23:12:31,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24222.11 MB 2025-02-14 23:12:31,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 23:12:31,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22190.05 MB 2025-02-14 23:12:31,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:12:31,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:12:31,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:12:31,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20772.63 MB 2025-02-14 23:12:31,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23014.48 MB 2025-02-14 23:12:31,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:12:31,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24222.11 MB 2025-02-14 23:12:31,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 23:12:31,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:12:31,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28558.76 MB 2025-02-14 23:12:31,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:12:31,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:12:31,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:12:31,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18883.09 MB 2025-02-14 23:12:31,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23014.48 MB 2025-02-14 23:12:31,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:12:31,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22334.67 MB 2025-02-14 23:12:31,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 23:12:31,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 23:12:31,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28558.76 MB 2025-02-14 23:12:31,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:12:31,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:12:31,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:12:31,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24548.02 MB 2025-02-14 23:12:31,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25315.03 MB 2025-02-14 23:12:31,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:12:31,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30356.28 MB 2025-02-14 23:12:31,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 23:12:31,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:12:31,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26022.81 MB 2025-02-14 23:12:31,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:12:31,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:12:31,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:12:31,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25727.91 MB 2025-02-14 23:12:31,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25955.76 MB 2025-02-14 23:12:31,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.84 MB 2025-02-14 23:12:31,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30771.51 MB 2025-02-14 23:12:31,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 23:12:31,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:12:31,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26134.87 MB 2025-02-14 23:12:31,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:12:31,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:12:31,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.96 seconds 2025-02-14 23:12:31,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14693.33 MB 2025-02-14 23:12:31,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26156.83 MB 2025-02-14 23:12:31,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11463.50 MB 2025-02-14 23:12:31,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41081.11 MB 2025-02-14 23:12:31,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 23:12:31,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10309.60 MB 2025-02-14 23:12:31,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26156.83 MB 2025-02-14 23:12:31,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:12:31,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:12:31,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:12:31,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26156.83 MB 2025-02-14 23:12:31,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19697.72 MB 2025-02-14 23:12:31,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6459.12 MB 2025-02-14 23:12:31,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30771.51 MB 2025-02-14 23:12:31,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 23:12:31,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:12:31,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28668.50 MB 2025-02-14 23:12:31,974 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:12:31,975 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 23:12:31,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:12:31,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:12:31,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 23:12:31,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:31,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19697.72 MB 2025-02-14 23:12:31,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28136.74 MB 2025-02-14 23:12:31,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:12:31,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30771.51 MB 2025-02-14 23:12:31,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41261.47 MB 2025-02-14 23:12:31,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 23:12:31,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28136.74 MB 2025-02-14 23:12:32,148 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:12:32,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:32,150 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:12:32,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:32,151 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:12:32,155 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:12:32,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:32,156 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:12:32,157 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 23:12:43,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:43,582 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:12:43,587 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:12:43,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:43,590 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 696, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:12:43,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:43,591 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 696, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:12:54,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:12:54,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:12:54,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.74 seconds 2025-02-14 23:12:54,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:54,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17818.55 MB 2025-02-14 23:12:54,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20281.65 MB 2025-02-14 23:12:54,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2463.11 MB 2025-02-14 23:12:54,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53846.47 MB 2025-02-14 23:12:54,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23720.89 MB 2025-02-14 23:12:54,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30125.59 MB 2025-02-14 23:12:54,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29101.86 MB 2025-02-14 23:12:54,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:12:54,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:12:54,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 23:12:54,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:54,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20281.65 MB 2025-02-14 23:12:54,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19397.19 MB 2025-02-14 23:12:54,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -884.46 MB 2025-02-14 23:12:54,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23720.89 MB 2025-02-14 23:12:54,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33237.76 MB 2025-02-14 23:12:54,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9516.88 MB 2025-02-14 23:12:54,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29130.66 MB 2025-02-14 23:12:56,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:12:56,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:12:56,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 23:12:56,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:56,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19397.19 MB 2025-02-14 23:12:56,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19928.03 MB 2025-02-14 23:12:56,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:12:56,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33237.76 MB 2025-02-14 23:12:56,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23746.05 MB 2025-02-14 23:12:56,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9491.71 MB 2025-02-14 23:12:56,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23907.61 MB 2025-02-14 23:12:56,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:12:56,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:12:56,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:12:56,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:56,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19928.03 MB 2025-02-14 23:12:56,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21817.56 MB 2025-02-14 23:12:56,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:12:56,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23746.05 MB 2025-02-14 23:12:56,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25633.49 MB 2025-02-14 23:12:56,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 23:12:56,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23234.99 MB 2025-02-14 23:12:56,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:12:56,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:12:56,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 23:12:56,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:56,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21817.56 MB 2025-02-14 23:12:56,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.42 MB 2025-02-14 23:12:56,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:12:56,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25633.49 MB 2025-02-14 23:12:56,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31297.90 MB 2025-02-14 23:12:56,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 23:12:56,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29603.70 MB 2025-02-14 23:12:56,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:12:56,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:12:56,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:12:56,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:56,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19928.03 MB 2025-02-14 23:12:56,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.42 MB 2025-02-14 23:12:56,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:12:56,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23746.05 MB 2025-02-14 23:12:56,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31297.90 MB 2025-02-14 23:12:56,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7551.84 MB 2025-02-14 23:12:56,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29603.70 MB 2025-02-14 23:12:56,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:12:56,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:12:56,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 23:12:56,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:56,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25592.96 MB 2025-02-14 23:12:56,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26359.96 MB 2025-02-14 23:12:56,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:12:56,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31297.90 MB 2025-02-14 23:12:56,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31715.23 MB 2025-02-14 23:12:56,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:12:56,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27067.75 MB 2025-02-14 23:12:56,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:12:56,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:12:56,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:12:56,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:56,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26772.85 MB 2025-02-14 23:12:56,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27000.98 MB 2025-02-14 23:12:56,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-14 23:12:56,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31715.23 MB 2025-02-14 23:12:56,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31715.23 MB 2025-02-14 23:12:56,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:12:56,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27179.54 MB 2025-02-14 23:12:56,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:12:56,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:12:56,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.17 seconds 2025-02-14 23:12:56,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:56,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15393.63 MB 2025-02-14 23:12:56,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27201.46 MB 2025-02-14 23:12:56,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11807.83 MB 2025-02-14 23:12:56,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53846.47 MB 2025-02-14 23:12:56,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31715.23 MB 2025-02-14 23:12:56,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22131.25 MB 2025-02-14 23:12:56,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27201.46 MB 2025-02-14 23:12:57,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:12:57,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:12:57,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:12:57,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:57,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27201.46 MB 2025-02-14 23:12:57,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20382.46 MB 2025-02-14 23:12:57,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6819.00 MB 2025-02-14 23:12:57,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31715.23 MB 2025-02-14 23:12:57,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31715.23 MB 2025-02-14 23:12:57,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:12:57,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29700.22 MB 2025-02-14 23:12:57,047 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 23:12:57,047 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:12:57,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:12:57,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:12:57,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:12:57,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:12:57,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20382.46 MB 2025-02-14 23:12:57,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28779.10 MB 2025-02-14 23:12:57,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-14 23:12:57,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31715.23 MB 2025-02-14 23:12:57,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40061.89 MB 2025-02-14 23:12:57,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 23:12:57,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28779.10 MB 2025-02-14 23:12:57,208 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 23:12:57,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:57,210 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:12:57,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:57,211 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:12:57,215 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:12:57,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:12:57,216 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:12:57,217 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:14:23,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:23,445 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:14:23,450 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:14:23,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:23,454 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 198, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:14:23,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:23,455 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 198, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:14:26,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:14:26,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:14:26,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.07 seconds 2025-02-14 23:14:26,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:26,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14348.40 MB 2025-02-14 23:14:26,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15049.11 MB 2025-02-14 23:14:26,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 700.71 MB 2025-02-14 23:14:26,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48408.56 MB 2025-02-14 23:14:26,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 23:14:26,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28642.90 MB 2025-02-14 23:14:26,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24046.27 MB 2025-02-14 23:14:26,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:14:26,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:14:26,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:14:26,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:26,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15049.11 MB 2025-02-14 23:14:26,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15388.61 MB 2025-02-14 23:14:26,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.49 MB 2025-02-14 23:14:26,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 23:14:26,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 23:14:26,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:14:26,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17851.53 MB 2025-02-14 23:14:27,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:14:27,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:14:27,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-14 23:14:27,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15388.61 MB 2025-02-14 23:14:27,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15651.37 MB 2025-02-14 23:14:27,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.77 MB 2025-02-14 23:14:27,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 23:14:27,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 23:14:27,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:14:27,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19644.23 MB 2025-02-14 23:14:27,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:14:27,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:14:27,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:14:27,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15651.31 MB 2025-02-14 23:14:27,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16586.40 MB 2025-02-14 23:14:27,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 935.09 MB 2025-02-14 23:14:27,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 23:14:27,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 23:14:27,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:14:27,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17288.03 MB 2025-02-14 23:14:27,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:14:27,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:14:27,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 23:14:27,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16586.40 MB 2025-02-14 23:14:27,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17696.15 MB 2025-02-14 23:14:27,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.75 MB 2025-02-14 23:14:27,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 23:14:27,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22106.08 MB 2025-02-14 23:14:27,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2340.42 MB 2025-02-14 23:14:27,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20442.64 MB 2025-02-14 23:14:27,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:14:27,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:14:27,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 23:14:27,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15651.31 MB 2025-02-14 23:14:27,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17696.15 MB 2025-02-14 23:14:27,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2044.84 MB 2025-02-14 23:14:27,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 23:14:27,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22106.08 MB 2025-02-14 23:14:27,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2340.42 MB 2025-02-14 23:14:27,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20442.64 MB 2025-02-14 23:14:27,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:14:27,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:14:27,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:14:27,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18455.25 MB 2025-02-14 23:14:27,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.92 MB 2025-02-14 23:14:27,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 379.67 MB 2025-02-14 23:14:27,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22106.08 MB 2025-02-14 23:14:27,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22311.60 MB 2025-02-14 23:14:27,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 205.52 MB 2025-02-14 23:14:27,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19187.48 MB 2025-02-14 23:14:27,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:14:27,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:14:27,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:14:27,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19039.31 MB 2025-02-14 23:14:27,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19259.92 MB 2025-02-14 23:14:27,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.61 MB 2025-02-14 23:14:27,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22311.60 MB 2025-02-14 23:14:27,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22311.60 MB 2025-02-14 23:14:27,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:14:27,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19311.13 MB 2025-02-14 23:14:27,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:14:27,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:14:27,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.25 seconds 2025-02-14 23:14:27,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13658.55 MB 2025-02-14 23:14:27,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19460.94 MB 2025-02-14 23:14:27,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5802.39 MB 2025-02-14 23:14:27,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48408.56 MB 2025-02-14 23:14:27,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22311.60 MB 2025-02-14 23:14:27,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26096.96 MB 2025-02-14 23:14:27,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19460.94 MB 2025-02-14 23:14:27,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:14:27,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:14:27,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 23:14:27,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:27,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14695.59 MB 2025-02-14 23:14:27,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17708.88 MB 2025-02-14 23:14:27,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3013.30 MB 2025-02-14 23:14:27,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22311.60 MB 2025-02-14 23:14:27,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22311.60 MB 2025-02-14 23:14:27,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:14:27,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18010.18 MB 2025-02-14 23:14:27,995 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 23:14:27,995 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 23:14:28,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:14:28,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:14:28,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:14:28,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:14:28,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17708.88 MB 2025-02-14 23:14:28,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26146.36 MB 2025-02-14 23:14:28,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 23:14:28,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22311.60 MB 2025-02-14 23:14:28,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30700.21 MB 2025-02-14 23:14:28,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 23:14:28,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26146.36 MB 2025-02-14 23:14:28,161 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 23:14:28,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:28,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:14:28,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:28,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:14:28,170 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:14:28,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:28,172 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:14:28,172 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 23:14:37,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:37,203 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:14:37,208 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:14:37,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:37,212 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2109, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:14:37,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:14:37,213 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2109, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:15:09,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:15:09,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:15:09,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.58 seconds 2025-02-14 23:15:09,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:09,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27664.56 MB 2025-02-14 23:15:09,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35128.32 MB 2025-02-14 23:15:09,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7463.76 MB 2025-02-14 23:15:09,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39088.82 MB 2025-02-14 23:15:09,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38159.78 MB 2025-02-14 23:15:09,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -929.04 MB 2025-02-14 23:15:09,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43931.51 MB 2025-02-14 23:15:09,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:15:09,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:15:09,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 23:15:09,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:09,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35128.32 MB 2025-02-14 23:15:09,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26742.93 MB 2025-02-14 23:15:09,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8385.39 MB 2025-02-14 23:15:09,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38159.78 MB 2025-02-14 23:15:09,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66244.84 MB 2025-02-14 23:15:09,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 28085.06 MB 2025-02-14 23:15:09,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54544.17 MB 2025-02-14 23:15:11,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:15:11,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:15:11,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 23:15:11,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:11,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26742.93 MB 2025-02-14 23:15:11,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27273.77 MB 2025-02-14 23:15:11,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:15:11,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66244.84 MB 2025-02-14 23:15:11,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29334.96 MB 2025-02-14 23:15:11,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36909.88 MB 2025-02-14 23:15:11,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31253.36 MB 2025-02-14 23:15:11,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:15:11,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:15:11,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:15:11,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:11,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27273.77 MB 2025-02-14 23:15:11,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29163.31 MB 2025-02-14 23:15:11,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:15:11,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29334.96 MB 2025-02-14 23:15:11,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32166.12 MB 2025-02-14 23:15:11,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 23:15:11,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30580.74 MB 2025-02-14 23:15:12,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:15:12,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:15:12,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 23:15:12,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:12,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29163.31 MB 2025-02-14 23:15:12,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31405.16 MB 2025-02-14 23:15:12,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:15:12,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32166.12 MB 2025-02-14 23:15:12,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38772.15 MB 2025-02-14 23:15:12,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:15:12,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36949.45 MB 2025-02-14 23:15:12,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:15:12,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:15:12,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:15:12,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:12,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27273.77 MB 2025-02-14 23:15:12,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31405.16 MB 2025-02-14 23:15:12,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:15:12,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29334.96 MB 2025-02-14 23:15:12,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38772.15 MB 2025-02-14 23:15:12,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 23:15:12,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36949.45 MB 2025-02-14 23:15:12,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:15:12,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:15:12,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 23:15:12,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:12,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32938.71 MB 2025-02-14 23:15:12,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33705.71 MB 2025-02-14 23:15:12,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:15:12,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38772.15 MB 2025-02-14 23:15:12,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39189.48 MB 2025-02-14 23:15:12,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:15:12,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34413.50 MB 2025-02-14 23:15:12,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:15:12,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:15:12,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:15:12,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:12,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34118.60 MB 2025-02-14 23:15:12,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34347.69 MB 2025-02-14 23:15:12,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.09 MB 2025-02-14 23:15:12,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39189.48 MB 2025-02-14 23:15:12,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39189.48 MB 2025-02-14 23:15:12,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:15:12,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34573.93 MB 2025-02-14 23:15:12,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:15:12,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:15:12,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.17 seconds 2025-02-14 23:15:12,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:12,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20316.63 MB 2025-02-14 23:15:12,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34548.05 MB 2025-02-14 23:15:12,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14231.42 MB 2025-02-14 23:15:12,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39088.82 MB 2025-02-14 23:15:12,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39189.48 MB 2025-02-14 23:15:12,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 100.66 MB 2025-02-14 23:15:12,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34573.93 MB 2025-02-14 23:15:12,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:15:12,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:15:12,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:15:12,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:12,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34548.05 MB 2025-02-14 23:15:12,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25310.10 MB 2025-02-14 23:15:12,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9237.95 MB 2025-02-14 23:15:12,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39189.48 MB 2025-02-14 23:15:12,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39189.48 MB 2025-02-14 23:15:12,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:15:12,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37050.81 MB 2025-02-14 23:15:12,670 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 23:15:12,670 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:15:12,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:15:12,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:15:12,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:15:12,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:15:12,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25310.10 MB 2025-02-14 23:15:12,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33719.40 MB 2025-02-14 23:15:12,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 23:15:12,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39189.48 MB 2025-02-14 23:15:12,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47548.73 MB 2025-02-14 23:15:12,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 23:15:12,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33719.40 MB 2025-02-14 23:15:12,831 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 23:15:12,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:15:12,832 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:15:12,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:15:12,833 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:15:12,838 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:15:12,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:15:12,839 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:15:12,839 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:16:35,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:35,780 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:16:35,785 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:16:35,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:35,790 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:16:35,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:35,791 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:16:38,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:16:38,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:16:38,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.80 seconds 2025-02-14 23:16:38,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:38,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-14 23:16:38,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-14 23:16:38,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-14 23:16:38,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55907.98 MB 2025-02-14 23:16:38,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 23:16:38,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35905.34 MB 2025-02-14 23:16:38,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23687.38 MB 2025-02-14 23:16:38,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:16:38,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:16:38,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:16:38,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:38,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-14 23:16:38,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15142.35 MB 2025-02-14 23:16:38,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.87 MB 2025-02-14 23:16:38,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 23:16:38,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20002.64 MB 2025-02-14 23:16:38,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:16:38,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.31 MB 2025-02-14 23:16:39,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:16:39,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:16:39,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 23:16:39,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15142.35 MB 2025-02-14 23:16:39,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15377.25 MB 2025-02-14 23:16:39,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 23:16:39,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20002.64 MB 2025-02-14 23:16:39,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 23:16:39,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 23:16:39,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19313.04 MB 2025-02-14 23:16:39,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:16:39,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:16:39,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:16:39,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-14 23:16:39,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16213.10 MB 2025-02-14 23:16:39,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 23:16:39,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 23:16:39,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 23:16:39,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:16:39,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16840.31 MB 2025-02-14 23:16:39,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:16:39,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:16:39,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 23:16:39,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16213.10 MB 2025-02-14 23:16:39,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-14 23:16:39,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 23:16:39,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 23:16:39,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21208.50 MB 2025-02-14 23:16:39,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1677.72 MB 2025-02-14 23:16:39,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19660.30 MB 2025-02-14 23:16:39,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:16:39,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:16:39,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 23:16:39,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.18 MB 2025-02-14 23:16:39,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.15 MB 2025-02-14 23:16:39,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 23:16:39,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 23:16:39,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21208.50 MB 2025-02-14 23:16:39,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1677.72 MB 2025-02-14 23:16:39,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19660.30 MB 2025-02-14 23:16:39,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:16:39,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:16:39,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 23:16:39,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17883.75 MB 2025-02-14 23:16:39,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18224.06 MB 2025-02-14 23:16:39,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-14 23:16:39,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21208.50 MB 2025-02-14 23:16:39,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 23:16:39,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 23:16:39,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18544.06 MB 2025-02-14 23:16:39,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:16:39,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:16:39,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:16:39,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18406.77 MB 2025-02-14 23:16:39,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18634.09 MB 2025-02-14 23:16:39,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.32 MB 2025-02-14 23:16:39,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 23:16:39,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 23:16:39,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:16:39,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18651.80 MB 2025-02-14 23:16:39,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:16:39,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:16:39,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.85 seconds 2025-02-14 23:16:39,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-14 23:16:39,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.95 MB 2025-02-14 23:16:39,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5242.59 MB 2025-02-14 23:16:39,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55907.98 MB 2025-02-14 23:16:39,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 23:16:39,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34517.02 MB 2025-02-14 23:16:39,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18834.95 MB 2025-02-14 23:16:39,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:16:39,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:16:39,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 23:16:39,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18834.95 MB 2025-02-14 23:16:39,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17541.83 MB 2025-02-14 23:16:39,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1293.11 MB 2025-02-14 23:16:39,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 23:16:39,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21390.95 MB 2025-02-14 23:16:39,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:16:39,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19070.03 MB 2025-02-14 23:16:39,928 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 23:16:39,928 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:16:39,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:16:39,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:16:39,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:16:39,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:16:39,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17541.83 MB 2025-02-14 23:16:39,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25972.23 MB 2025-02-14 23:16:39,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 23:16:39,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21390.95 MB 2025-02-14 23:16:39,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29771.17 MB 2025-02-14 23:16:39,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 23:16:39,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25972.23 MB 2025-02-14 23:16:40,096 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 23:16:40,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:40,097 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:16:40,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:40,098 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:16:40,103 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:16:40,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:40,104 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:16:40,104 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:16:48,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:48,372 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:16:48,377 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:16:48,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:48,380 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1742, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:16:48,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:16:48,381 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1742, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:17:15,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:17:15,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:17:15,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.85 seconds 2025-02-14 23:17:15,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:15,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25107.24 MB 2025-02-14 23:17:15,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31272.87 MB 2025-02-14 23:17:15,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6165.63 MB 2025-02-14 23:17:15,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38151.39 MB 2025-02-14 23:17:15,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36849.06 MB 2025-02-14 23:17:15,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1302.33 MB 2025-02-14 23:17:15,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40241.73 MB 2025-02-14 23:17:15,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:17:15,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:17:15,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 23:17:15,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:15,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31272.87 MB 2025-02-14 23:17:15,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24833.97 MB 2025-02-14 23:17:15,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6438.90 MB 2025-02-14 23:17:15,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36849.06 MB 2025-02-14 23:17:15,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57461.96 MB 2025-02-14 23:17:15,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20612.91 MB 2025-02-14 23:17:15,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48616.60 MB 2025-02-14 23:17:17,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:17:17,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:17:17,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 23:17:17,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24833.97 MB 2025-02-14 23:17:17,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25364.81 MB 2025-02-14 23:17:17,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:17:17,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57461.96 MB 2025-02-14 23:17:17,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32099.01 MB 2025-02-14 23:17:17,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25362.96 MB 2025-02-14 23:17:17,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.35 MB 2025-02-14 23:17:17,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:17:17,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:17:17,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:17:17,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-14 23:17:17,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27254.34 MB 2025-02-14 23:17:17,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:17:17,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32099.01 MB 2025-02-14 23:17:17,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32099.01 MB 2025-02-14 23:17:17,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:17:17,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28671.77 MB 2025-02-14 23:17:17,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:17:17,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:17:17,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:17:17,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27254.34 MB 2025-02-14 23:17:17,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-14 23:17:17,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:17:17,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32099.01 MB 2025-02-14 23:17:17,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37289.46 MB 2025-02-14 23:17:17,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 23:17:17,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-14 23:17:17,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:17:17,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:17:17,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:17:17,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-14 23:17:17,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-14 23:17:17,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:17:17,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32099.01 MB 2025-02-14 23:17:17,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37289.46 MB 2025-02-14 23:17:17,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 23:17:17,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-14 23:17:17,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:17:17,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:17:17,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:17:17,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31029.74 MB 2025-02-14 23:17:17,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31796.74 MB 2025-02-14 23:17:17,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:17:17,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37289.46 MB 2025-02-14 23:17:17,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37706.79 MB 2025-02-14 23:17:17,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:17:17,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32504.53 MB 2025-02-14 23:17:17,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:17:17,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:17:17,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:17:17,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32209.63 MB 2025-02-14 23:17:17,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32437.73 MB 2025-02-14 23:17:17,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 23:17:17,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37706.79 MB 2025-02-14 23:17:17,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37706.79 MB 2025-02-14 23:17:17,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:17:17,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32640.13 MB 2025-02-14 23:17:17,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:17:17,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:17:17,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.30 seconds 2025-02-14 23:17:17,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19037.97 MB 2025-02-14 23:17:17,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32637.75 MB 2025-02-14 23:17:17,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13599.77 MB 2025-02-14 23:17:17,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38151.39 MB 2025-02-14 23:17:17,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37706.79 MB 2025-02-14 23:17:17,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -444.60 MB 2025-02-14 23:17:17,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32640.13 MB 2025-02-14 23:17:17,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:17:17,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:17:17,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:17:17,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32637.75 MB 2025-02-14 23:17:17,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24026.45 MB 2025-02-14 23:17:17,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8611.30 MB 2025-02-14 23:17:17,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37706.79 MB 2025-02-14 23:17:17,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37706.79 MB 2025-02-14 23:17:17,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:17:17,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35136.67 MB 2025-02-14 23:17:17,975 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 23:17:17,975 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:17:17,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:17:17,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:17:17,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:17:17,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:17:17,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24026.45 MB 2025-02-14 23:17:17,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32421.66 MB 2025-02-14 23:17:17,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 23:17:17,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37706.79 MB 2025-02-14 23:17:17,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46053.46 MB 2025-02-14 23:17:17,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 23:17:17,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32421.66 MB 2025-02-14 23:17:18,136 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 23:17:18,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:17:18,137 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:17:18,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:17:18,138 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:17:18,143 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:17:18,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:17:18,144 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:17:18,144 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:18:23,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:18:23,955 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:18:23,963 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:18:23,970 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:18:23,970 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:18:23,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:18:23,972 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:18:26,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:18:26,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:18:26,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.19 seconds 2025-02-14 23:18:26,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:26,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13916.38 MB 2025-02-14 23:18:26,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14397.67 MB 2025-02-14 23:18:26,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.30 MB 2025-02-14 23:18:26,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54400.12 MB 2025-02-14 23:18:26,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 23:18:26,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33925.63 MB 2025-02-14 23:18:26,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23387.75 MB 2025-02-14 23:18:26,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:18:26,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:18:26,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:18:26,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:26,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14397.67 MB 2025-02-14 23:18:26,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14588.72 MB 2025-02-14 23:18:26,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.05 MB 2025-02-14 23:18:26,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 23:18:26,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 23:18:26,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:18:26,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16252.04 MB 2025-02-14 23:18:26,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:18:26,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:18:26,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.65 seconds 2025-02-14 23:18:26,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:26,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14588.72 MB 2025-02-14 23:18:26,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14761.24 MB 2025-02-14 23:18:26,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-14 23:18:26,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 23:18:26,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 23:18:26,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:18:26,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18758.37 MB 2025-02-14 23:18:26,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:18:26,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:18:26,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 23:18:26,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:26,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-14 23:18:26,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15375.13 MB 2025-02-14 23:18:26,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-14 23:18:26,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 23:18:26,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 23:18:26,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:18:26,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15835.80 MB 2025-02-14 23:18:26,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:18:26,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:18:26,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 23:18:26,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:26,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15375.13 MB 2025-02-14 23:18:26,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-14 23:18:26,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-14 23:18:26,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 23:18:26,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 23:18:26,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:18:26,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-14 23:18:26,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:18:26,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:18:26,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 23:18:26,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:26,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-14 23:18:26,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-14 23:18:26,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-14 23:18:26,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 23:18:26,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20474.49 MB 2025-02-14 23:18:26,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:18:26,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-14 23:18:27,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:18:27,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:18:27,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 23:18:27,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:27,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16602.18 MB 2025-02-14 23:18:27,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16851.45 MB 2025-02-14 23:18:27,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-14 23:18:27,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20474.49 MB 2025-02-14 23:18:27,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20606.62 MB 2025-02-14 23:18:27,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 23:18:27,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17094.11 MB 2025-02-14 23:18:27,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:18:27,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:18:27,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:18:27,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:27,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16985.65 MB 2025-02-14 23:18:27,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17190.81 MB 2025-02-14 23:18:27,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.16 MB 2025-02-14 23:18:27,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20606.62 MB 2025-02-14 23:18:27,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20610.81 MB 2025-02-14 23:18:27,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 23:18:27,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17190.81 MB 2025-02-14 23:18:27,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:18:27,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:18:27,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.08 seconds 2025-02-14 23:18:27,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:27,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13442.54 MB 2025-02-14 23:18:27,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17391.64 MB 2025-02-14 23:18:27,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3949.10 MB 2025-02-14 23:18:27,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54400.12 MB 2025-02-14 23:18:27,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20610.81 MB 2025-02-14 23:18:27,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33789.31 MB 2025-02-14 23:18:27,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17391.64 MB 2025-02-14 23:18:27,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:18:27,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:18:27,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 23:18:27,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:27,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17391.64 MB 2025-02-14 23:18:27,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17168.91 MB 2025-02-14 23:18:27,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -222.72 MB 2025-02-14 23:18:27,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20610.81 MB 2025-02-14 23:18:27,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20610.81 MB 2025-02-14 23:18:27,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:18:27,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18796.45 MB 2025-02-14 23:18:27,361 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 23:18:27,362 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:18:27,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:18:27,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:18:27,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:18:27,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:18:27,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17168.91 MB 2025-02-14 23:18:27,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25598.04 MB 2025-02-14 23:18:27,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 23:18:27,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20610.81 MB 2025-02-14 23:18:27,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31086.08 MB 2025-02-14 23:18:27,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 23:18:27,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25598.04 MB 2025-02-14 23:18:27,617 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 23:18:27,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:18:27,620 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:18:27,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:18:27,622 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:18:27,629 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:18:27,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:18:27,631 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:18:27,632 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:19:28,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:19:28,060 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:19:28,065 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:19:28,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:19:28,069 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1627, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:19:28,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:19:28,070 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1627, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:19:52,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:19:52,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:19:52,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.90 seconds 2025-02-14 23:19:52,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:52,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24305.90 MB 2025-02-14 23:19:52,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30064.68 MB 2025-02-14 23:19:52,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5758.78 MB 2025-02-14 23:19:52,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39466.30 MB 2025-02-14 23:19:52,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36444.31 MB 2025-02-14 23:19:52,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3022.00 MB 2025-02-14 23:19:52,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38987.41 MB 2025-02-14 23:19:53,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:19:53,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:19:53,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:19:53,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:53,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30064.68 MB 2025-02-14 23:19:53,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24236.12 MB 2025-02-14 23:19:53,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5828.57 MB 2025-02-14 23:19:53,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36444.31 MB 2025-02-14 23:19:53,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46282.05 MB 2025-02-14 23:19:53,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9837.74 MB 2025-02-14 23:19:53,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40945.26 MB 2025-02-14 23:19:54,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:19:54,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:19:54,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 23:19:54,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:54,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24236.12 MB 2025-02-14 23:19:54,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24766.96 MB 2025-02-14 23:19:54,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:19:54,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46282.05 MB 2025-02-14 23:19:54,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 23:19:54,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14180.94 MB 2025-02-14 23:19:54,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28745.50 MB 2025-02-14 23:19:54,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:19:54,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:19:54,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:19:54,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:54,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24766.96 MB 2025-02-14 23:19:54,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26656.49 MB 2025-02-14 23:19:54,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:19:54,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 23:19:54,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 23:19:54,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:19:54,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28073.92 MB 2025-02-14 23:19:55,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:19:55,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:19:55,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:19:55,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:55,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26656.49 MB 2025-02-14 23:19:55,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28898.35 MB 2025-02-14 23:19:55,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:19:55,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 23:19:55,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36819.70 MB 2025-02-14 23:19:55,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 23:19:55,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34442.63 MB 2025-02-14 23:19:55,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:19:55,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:19:55,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:19:55,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:55,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24766.96 MB 2025-02-14 23:19:55,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28898.35 MB 2025-02-14 23:19:55,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:19:55,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 23:19:55,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36819.70 MB 2025-02-14 23:19:55,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 23:19:55,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34442.63 MB 2025-02-14 23:19:55,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:19:55,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:19:55,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:19:55,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:55,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30431.89 MB 2025-02-14 23:19:55,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31198.89 MB 2025-02-14 23:19:55,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:19:55,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36819.70 MB 2025-02-14 23:19:55,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37234.93 MB 2025-02-14 23:19:55,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:19:55,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31906.68 MB 2025-02-14 23:19:55,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:19:55,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:19:55,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:19:55,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:55,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31611.78 MB 2025-02-14 23:19:55,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31843.43 MB 2025-02-14 23:19:55,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.65 MB 2025-02-14 23:19:55,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37234.93 MB 2025-02-14 23:19:55,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37234.93 MB 2025-02-14 23:19:55,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:19:55,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32054.76 MB 2025-02-14 23:19:55,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:19:55,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:19:55,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.31 seconds 2025-02-14 23:19:55,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:55,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18637.30 MB 2025-02-14 23:19:55,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32044.51 MB 2025-02-14 23:19:55,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13407.20 MB 2025-02-14 23:19:55,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39466.30 MB 2025-02-14 23:19:55,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37234.93 MB 2025-02-14 23:19:55,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2231.37 MB 2025-02-14 23:19:55,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32054.76 MB 2025-02-14 23:19:55,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:19:55,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:19:55,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:19:55,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:55,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32044.51 MB 2025-02-14 23:19:55,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23641.69 MB 2025-02-14 23:19:55,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8402.81 MB 2025-02-14 23:19:55,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37234.93 MB 2025-02-14 23:19:55,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37234.93 MB 2025-02-14 23:19:55,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:19:55,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34556.17 MB 2025-02-14 23:19:55,670 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:19:55,670 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:19:55,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:19:55,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:19:55,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:19:55,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:19:55,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23641.69 MB 2025-02-14 23:19:55,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32080.72 MB 2025-02-14 23:19:55,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:19:55,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37234.93 MB 2025-02-14 23:19:55,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45625.64 MB 2025-02-14 23:19:55,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:19:55,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32080.72 MB 2025-02-14 23:19:55,832 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:19:55,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:19:55,834 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:19:55,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:19:55,834 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:19:55,839 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:19:55,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:19:55,840 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:19:55,840 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:20:50,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:20:50,615 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:20:50,620 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:20:50,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:20:50,624 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1077, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:20:50,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:20:50,625 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1077, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:21:07,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:21:07,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:21:07,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.54 seconds 2025-02-14 23:21:07,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:07,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20473.41 MB 2025-02-14 23:21:07,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24284.86 MB 2025-02-14 23:21:07,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3811.44 MB 2025-02-14 23:21:07,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58210.65 MB 2025-02-14 23:21:07,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26115.83 MB 2025-02-14 23:21:07,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32094.81 MB 2025-02-14 23:21:07,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33116.49 MB 2025-02-14 23:21:07,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:21:07,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:21:07,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:21:07,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:07,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24284.86 MB 2025-02-14 23:21:07,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21377.89 MB 2025-02-14 23:21:07,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2906.97 MB 2025-02-14 23:21:07,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26115.83 MB 2025-02-14 23:21:07,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43985.67 MB 2025-02-14 23:21:07,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17869.83 MB 2025-02-14 23:21:07,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35755.08 MB 2025-02-14 23:21:09,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:21:09,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:21:09,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 23:21:09,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21377.89 MB 2025-02-14 23:21:09,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21908.73 MB 2025-02-14 23:21:09,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:21:09,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43985.67 MB 2025-02-14 23:21:09,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25136.46 MB 2025-02-14 23:21:09,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18849.20 MB 2025-02-14 23:21:09,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25888.31 MB 2025-02-14 23:21:09,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:21:09,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:21:09,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:21:09,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21908.73 MB 2025-02-14 23:21:09,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23798.26 MB 2025-02-14 23:21:09,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:21:09,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25136.46 MB 2025-02-14 23:21:09,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27023.90 MB 2025-02-14 23:21:09,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 23:21:09,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25215.69 MB 2025-02-14 23:21:09,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:21:09,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:21:09,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:21:09,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23798.26 MB 2025-02-14 23:21:09,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26040.12 MB 2025-02-14 23:21:09,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:21:09,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27023.90 MB 2025-02-14 23:21:09,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33629.93 MB 2025-02-14 23:21:09,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:21:09,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31584.40 MB 2025-02-14 23:21:09,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:21:09,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:21:09,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:21:09,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21908.73 MB 2025-02-14 23:21:09,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26040.12 MB 2025-02-14 23:21:09,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:21:09,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25136.46 MB 2025-02-14 23:21:09,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33629.93 MB 2025-02-14 23:21:09,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 23:21:09,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31584.40 MB 2025-02-14 23:21:09,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:21:09,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:21:09,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:21:09,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27573.66 MB 2025-02-14 23:21:09,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28340.66 MB 2025-02-14 23:21:09,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:21:09,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33629.93 MB 2025-02-14 23:21:09,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34047.26 MB 2025-02-14 23:21:09,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:21:09,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29048.45 MB 2025-02-14 23:21:09,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:21:09,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:21:09,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:21:09,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28753.55 MB 2025-02-14 23:21:09,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28981.48 MB 2025-02-14 23:21:09,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.93 MB 2025-02-14 23:21:09,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34047.26 MB 2025-02-14 23:21:09,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34047.26 MB 2025-02-14 23:21:09,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:21:09,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29199.83 MB 2025-02-14 23:21:09,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:21:09,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:21:09,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.98 seconds 2025-02-14 23:21:09,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16721.06 MB 2025-02-14 23:21:09,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29181.91 MB 2025-02-14 23:21:09,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12460.85 MB 2025-02-14 23:21:09,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58210.65 MB 2025-02-14 23:21:09,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34047.26 MB 2025-02-14 23:21:09,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24163.39 MB 2025-02-14 23:21:09,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29199.83 MB 2025-02-14 23:21:09,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:21:09,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:21:09,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:21:09,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29181.91 MB 2025-02-14 23:21:09,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21715.55 MB 2025-02-14 23:21:09,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7466.37 MB 2025-02-14 23:21:09,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34047.26 MB 2025-02-14 23:21:09,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34047.26 MB 2025-02-14 23:21:09,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:21:09,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31685.59 MB 2025-02-14 23:21:09,897 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 23:21:09,898 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:21:09,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:21:09,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:21:09,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:21:09,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:21:09,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21715.55 MB 2025-02-14 23:21:09,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30128.98 MB 2025-02-14 23:21:09,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.43 MB 2025-02-14 23:21:09,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34047.26 MB 2025-02-14 23:21:09,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44501.57 MB 2025-02-14 23:21:09,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 23:21:09,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30128.98 MB 2025-02-14 23:21:10,063 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 23:21:10,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:21:10,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:21:10,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:21:10,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:21:10,070 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:21:10,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:21:10,071 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:21:10,071 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:22:47,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:22:47,347 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:22:47,352 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:22:47,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:22:47,357 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1538, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:22:47,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:22:47,358 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1538, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:23:10,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:23:10,991 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:23:10,991 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.62 seconds 2025-02-14 23:23:10,991 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:10,991 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23685.74 MB 2025-02-14 23:23:10,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29128.63 MB 2025-02-14 23:23:10,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5442.90 MB 2025-02-14 23:23:10,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52865.01 MB 2025-02-14 23:23:10,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36104.57 MB 2025-02-14 23:23:10,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16760.44 MB 2025-02-14 23:23:10,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38140.75 MB 2025-02-14 23:23:11,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:23:11,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:23:11,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:23:11,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:11,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29128.63 MB 2025-02-14 23:23:11,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23773.43 MB 2025-02-14 23:23:11,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5355.20 MB 2025-02-14 23:23:11,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36104.57 MB 2025-02-14 23:23:11,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44715.47 MB 2025-02-14 23:23:11,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8610.91 MB 2025-02-14 23:23:11,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41017.30 MB 2025-02-14 23:23:12,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:23:12,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:23:12,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 23:23:12,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:12,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23773.43 MB 2025-02-14 23:23:12,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24304.27 MB 2025-02-14 23:23:12,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:23:12,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44715.47 MB 2025-02-14 23:23:12,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 23:23:12,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14055.11 MB 2025-02-14 23:23:12,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28282.82 MB 2025-02-14 23:23:12,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:23:12,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:23:12,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:23:12,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:12,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24304.27 MB 2025-02-14 23:23:12,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26193.81 MB 2025-02-14 23:23:12,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:23:12,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 23:23:12,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30660.36 MB 2025-02-14 23:23:12,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:23:12,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27611.24 MB 2025-02-14 23:23:13,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:23:13,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:23:13,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:23:13,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:13,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26193.81 MB 2025-02-14 23:23:13,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28435.66 MB 2025-02-14 23:23:13,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:23:13,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 23:23:13,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 23:23:13,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:23:13,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33979.95 MB 2025-02-14 23:23:13,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:23:13,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:23:13,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:23:13,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:13,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24304.27 MB 2025-02-14 23:23:13,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28435.66 MB 2025-02-14 23:23:13,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:23:13,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30660.36 MB 2025-02-14 23:23:13,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36322.67 MB 2025-02-14 23:23:13,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:23:13,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33979.95 MB 2025-02-14 23:23:13,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:23:13,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:23:13,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:23:13,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:13,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29969.21 MB 2025-02-14 23:23:13,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30736.21 MB 2025-02-14 23:23:13,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:23:13,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36322.67 MB 2025-02-14 23:23:13,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 23:23:13,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:23:13,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31444.00 MB 2025-02-14 23:23:13,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:23:13,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:23:13,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:23:13,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:13,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31149.10 MB 2025-02-14 23:23:13,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31377.17 MB 2025-02-14 23:23:13,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 23:23:13,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36737.91 MB 2025-02-14 23:23:13,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 23:23:13,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:23:13,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31610.72 MB 2025-02-14 23:23:13,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:23:13,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:23:13,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.03 seconds 2025-02-14 23:23:13,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:13,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18327.22 MB 2025-02-14 23:23:13,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31577.16 MB 2025-02-14 23:23:13,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13249.94 MB 2025-02-14 23:23:13,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52865.01 MB 2025-02-14 23:23:13,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 23:23:13,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16127.10 MB 2025-02-14 23:23:13,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31610.72 MB 2025-02-14 23:23:13,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:23:13,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:23:13,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:23:13,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:13,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31577.16 MB 2025-02-14 23:23:13,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23315.34 MB 2025-02-14 23:23:13,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8261.82 MB 2025-02-14 23:23:13,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36737.91 MB 2025-02-14 23:23:13,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36737.91 MB 2025-02-14 23:23:13,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:23:13,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34075.32 MB 2025-02-14 23:23:13,683 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 23:23:13,684 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:23:13,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:23:13,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:23:13,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:23:13,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:23:13,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23315.34 MB 2025-02-14 23:23:13,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31708.62 MB 2025-02-14 23:23:13,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 23:23:13,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36737.91 MB 2025-02-14 23:23:13,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45084.57 MB 2025-02-14 23:23:13,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 23:23:13,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31708.62 MB 2025-02-14 23:23:13,850 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 23:23:13,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:23:13,851 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:23:13,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:23:13,852 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:23:13,858 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:23:13,859 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:23:13,859 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:23:13,860 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:24:18,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:24:18,088 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:24:18,093 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:24:18,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:24:18,097 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2418, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:24:18,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:24:18,098 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2418, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:24:55,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:24:55,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:24:55,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.21 seconds 2025-02-14 23:24:55,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:55,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29817.72 MB 2025-02-14 23:24:55,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38374.88 MB 2025-02-14 23:24:55,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8557.17 MB 2025-02-14 23:24:55,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61991.81 MB 2025-02-14 23:24:55,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43580.92 MB 2025-02-14 23:24:55,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18410.90 MB 2025-02-14 23:24:55,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47216.32 MB 2025-02-14 23:24:55,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:24:55,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:24:55,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:24:55,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:55,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38374.88 MB 2025-02-14 23:24:55,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28348.28 MB 2025-02-14 23:24:55,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10026.61 MB 2025-02-14 23:24:55,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43580.92 MB 2025-02-14 23:24:55,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 72299.32 MB 2025-02-14 23:24:55,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 28718.40 MB 2025-02-14 23:24:55,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62447.00 MB 2025-02-14 23:24:57,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:24:57,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:24:57,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 23:24:57,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:57,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28348.28 MB 2025-02-14 23:24:57,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28879.12 MB 2025-02-14 23:24:57,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:24:57,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72299.32 MB 2025-02-14 23:24:57,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32157.73 MB 2025-02-14 23:24:57,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40141.59 MB 2025-02-14 23:24:57,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32858.71 MB 2025-02-14 23:24:57,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:24:57,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:24:57,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:24:57,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:57,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28879.12 MB 2025-02-14 23:24:57,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30768.26 MB 2025-02-14 23:24:57,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.14 MB 2025-02-14 23:24:57,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32157.73 MB 2025-02-14 23:24:57,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34045.17 MB 2025-02-14 23:24:57,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 23:24:57,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32185.69 MB 2025-02-14 23:24:57,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:24:57,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:24:57,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:24:57,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:57,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30768.26 MB 2025-02-14 23:24:57,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33010.12 MB 2025-02-14 23:24:57,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:24:57,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34045.17 MB 2025-02-14 23:24:57,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40179.34 MB 2025-02-14 23:24:57,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:24:57,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38554.40 MB 2025-02-14 23:24:57,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:24:57,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:24:57,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:24:57,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:57,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28879.12 MB 2025-02-14 23:24:57,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33010.12 MB 2025-02-14 23:24:57,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.00 MB 2025-02-14 23:24:57,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32157.73 MB 2025-02-14 23:24:57,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40179.34 MB 2025-02-14 23:24:57,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 23:24:57,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38554.40 MB 2025-02-14 23:24:57,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:24:57,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:24:57,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:24:57,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:57,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34543.66 MB 2025-02-14 23:24:57,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35310.66 MB 2025-02-14 23:24:57,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:24:57,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40179.34 MB 2025-02-14 23:24:57,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-14 23:24:57,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:24:57,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36018.45 MB 2025-02-14 23:24:57,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:24:57,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:24:57,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:24:57,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:57,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35723.55 MB 2025-02-14 23:24:57,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35952.39 MB 2025-02-14 23:24:57,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-14 23:24:57,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40594.57 MB 2025-02-14 23:24:57,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-14 23:24:57,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:24:57,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36168.61 MB 2025-02-14 23:24:57,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:24:57,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:24:57,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.75 seconds 2025-02-14 23:24:57,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:57,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-14 23:24:57,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36153.14 MB 2025-02-14 23:24:57,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14759.93 MB 2025-02-14 23:24:57,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57711.53 MB 2025-02-14 23:24:57,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-14 23:24:57,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17116.95 MB 2025-02-14 23:24:57,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36168.61 MB 2025-02-14 23:24:58,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:24:58,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:24:58,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:24:58,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:58,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36153.14 MB 2025-02-14 23:24:58,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26392.65 MB 2025-02-14 23:24:58,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9760.49 MB 2025-02-14 23:24:58,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40594.57 MB 2025-02-14 23:24:58,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40594.57 MB 2025-02-14 23:24:58,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:24:58,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38660.81 MB 2025-02-14 23:24:58,139 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 23:24:58,140 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:24:58,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:24:58,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:24:58,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:24:58,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:24:58,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26392.65 MB 2025-02-14 23:24:58,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34818.83 MB 2025-02-14 23:24:58,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 23:24:58,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40594.57 MB 2025-02-14 23:24:58,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48970.60 MB 2025-02-14 23:24:58,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 23:24:58,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34818.83 MB 2025-02-14 23:24:58,305 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 23:24:58,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:24:58,307 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:24:58,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:24:58,308 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:24:58,314 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:24:58,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:24:58,315 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:24:58,315 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:25:52,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:25:52,126 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:25:52,131 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:25:52,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:25:52,134 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1344, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:25:52,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:25:52,135 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1344, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:26:12,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:26:12,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:26:12,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.76 seconds 2025-02-14 23:26:12,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:12,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22333.91 MB 2025-02-14 23:26:12,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27090.25 MB 2025-02-14 23:26:12,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4756.34 MB 2025-02-14 23:26:12,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57346.62 MB 2025-02-14 23:26:12,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35404.12 MB 2025-02-14 23:26:12,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21942.50 MB 2025-02-14 23:26:12,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36108.64 MB 2025-02-14 23:26:12,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:26:12,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:26:12,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:26:12,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:12,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27090.25 MB 2025-02-14 23:26:12,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22764.89 MB 2025-02-14 23:26:12,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4325.37 MB 2025-02-14 23:26:12,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35404.12 MB 2025-02-14 23:26:12,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46063.94 MB 2025-02-14 23:26:12,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10659.82 MB 2025-02-14 23:26:12,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41226.94 MB 2025-02-14 23:26:14,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:26:14,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:26:14,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:26:14,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:14,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22764.89 MB 2025-02-14 23:26:14,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23295.73 MB 2025-02-14 23:26:14,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:26:14,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46063.94 MB 2025-02-14 23:26:14,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30647.78 MB 2025-02-14 23:26:14,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15416.16 MB 2025-02-14 23:26:14,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27274.28 MB 2025-02-14 23:26:14,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:26:14,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:26:14,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:26:14,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:14,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23295.73 MB 2025-02-14 23:26:14,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25185.26 MB 2025-02-14 23:26:14,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:26:14,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30647.78 MB 2025-02-14 23:26:14,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30647.78 MB 2025-02-14 23:26:14,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:26:14,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26602.69 MB 2025-02-14 23:26:15,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:26:15,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:26:15,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 23:26:15,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:15,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25185.26 MB 2025-02-14 23:26:15,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27427.12 MB 2025-02-14 23:26:15,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:26:15,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30647.78 MB 2025-02-14 23:26:15,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35366.37 MB 2025-02-14 23:26:15,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 23:26:15,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32971.40 MB 2025-02-14 23:26:15,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:26:15,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:26:15,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 23:26:15,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:15,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23295.73 MB 2025-02-14 23:26:15,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27427.12 MB 2025-02-14 23:26:15,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:26:15,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30647.78 MB 2025-02-14 23:26:15,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35366.37 MB 2025-02-14 23:26:15,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 23:26:15,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32971.40 MB 2025-02-14 23:26:15,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:26:15,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:26:15,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 23:26:15,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:15,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28960.66 MB 2025-02-14 23:26:15,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29727.66 MB 2025-02-14 23:26:15,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:26:15,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35366.37 MB 2025-02-14 23:26:15,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 23:26:15,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:26:15,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30435.45 MB 2025-02-14 23:26:15,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:26:15,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:26:15,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 23:26:15,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:15,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30140.55 MB 2025-02-14 23:26:15,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30369.05 MB 2025-02-14 23:26:15,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.49 MB 2025-02-14 23:26:15,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 23:26:15,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 23:26:15,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:26:15,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30613.92 MB 2025-02-14 23:26:15,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:26:15,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:26:15,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.32 seconds 2025-02-14 23:26:15,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:15,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17651.31 MB 2025-02-14 23:26:15,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30569.45 MB 2025-02-14 23:26:15,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12918.15 MB 2025-02-14 23:26:15,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57346.62 MB 2025-02-14 23:26:15,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 23:26:15,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21562.92 MB 2025-02-14 23:26:15,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30613.92 MB 2025-02-14 23:26:15,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:26:15,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:26:15,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 23:26:15,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:15,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30569.45 MB 2025-02-14 23:26:15,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22645.78 MB 2025-02-14 23:26:15,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7923.67 MB 2025-02-14 23:26:15,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 23:26:15,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 23:26:15,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:26:15,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33073.12 MB 2025-02-14 23:26:15,773 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 23:26:15,774 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:26:15,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:26:15,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:26:15,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:26:15,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:26:15,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22645.78 MB 2025-02-14 23:26:15,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31056.61 MB 2025-02-14 23:26:15,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 23:26:15,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 23:26:15,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44147.15 MB 2025-02-14 23:26:15,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 23:26:15,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31056.61 MB 2025-02-14 23:26:15,986 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 23:26:15,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:26:15,989 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:26:15,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:26:15,991 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:26:15,998 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:26:16,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:26:16,000 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:26:16,001 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:27:10,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:27:10,174 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:27:10,179 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:27:10,183 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:27:10,183 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1340, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:27:10,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:27:10,184 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1340, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:27:30,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:27:30,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:27:30,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.64 seconds 2025-02-14 23:27:30,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:30,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22306.04 MB 2025-02-14 23:27:30,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27048.23 MB 2025-02-14 23:27:30,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.18 MB 2025-02-14 23:27:30,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52510.59 MB 2025-02-14 23:27:30,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35387.34 MB 2025-02-14 23:27:30,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17123.25 MB 2025-02-14 23:27:30,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35854.28 MB 2025-02-14 23:27:30,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:27:30,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:27:30,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:27:30,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:30,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.23 MB 2025-02-14 23:27:30,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.09 MB 2025-02-14 23:27:30,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4304.13 MB 2025-02-14 23:27:30,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35387.34 MB 2025-02-14 23:27:30,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45298.48 MB 2025-02-14 23:27:30,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9911.14 MB 2025-02-14 23:27:30,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40644.24 MB 2025-02-14 23:27:32,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:27:32,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:27:32,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:27:32,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:32,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.09 MB 2025-02-14 23:27:32,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23274.93 MB 2025-02-14 23:27:32,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:27:32,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45298.48 MB 2025-02-14 23:27:32,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30643.59 MB 2025-02-14 23:27:32,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14654.90 MB 2025-02-14 23:27:32,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27253.48 MB 2025-02-14 23:27:32,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:27:32,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:27:32,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:27:32,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:32,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 23:27:32,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25164.47 MB 2025-02-14 23:27:32,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:27:32,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30643.59 MB 2025-02-14 23:27:32,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30643.59 MB 2025-02-14 23:27:32,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:27:32,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26581.90 MB 2025-02-14 23:27:33,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:27:33,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:27:33,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:27:33,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:33,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25164.47 MB 2025-02-14 23:27:33,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-14 23:27:33,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:27:33,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30643.59 MB 2025-02-14 23:27:33,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35362.18 MB 2025-02-14 23:27:33,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 23:27:33,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-14 23:27:33,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:27:33,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:27:33,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:27:33,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:33,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 23:27:33,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-14 23:27:33,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:27:33,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30643.59 MB 2025-02-14 23:27:33,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35362.18 MB 2025-02-14 23:27:33,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 23:27:33,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-14 23:27:33,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:27:33,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:27:33,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:27:33,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:33,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.87 MB 2025-02-14 23:27:33,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29706.87 MB 2025-02-14 23:27:33,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:27:33,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35362.18 MB 2025-02-14 23:27:33,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35777.41 MB 2025-02-14 23:27:33,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:27:33,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30414.66 MB 2025-02-14 23:27:33,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:27:33,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:27:33,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:27:33,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:33,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30119.76 MB 2025-02-14 23:27:33,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30346.54 MB 2025-02-14 23:27:33,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.79 MB 2025-02-14 23:27:33,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35777.41 MB 2025-02-14 23:27:33,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35777.41 MB 2025-02-14 23:27:33,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:27:33,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30547.15 MB 2025-02-14 23:27:33,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:27:33,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:27:33,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.08 seconds 2025-02-14 23:27:33,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:33,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-14 23:27:33,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30547.03 MB 2025-02-14 23:27:33,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12909.65 MB 2025-02-14 23:27:33,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52510.59 MB 2025-02-14 23:27:33,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35777.41 MB 2025-02-14 23:27:33,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16733.18 MB 2025-02-14 23:27:33,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30547.15 MB 2025-02-14 23:27:33,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:27:33,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:27:33,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:27:33,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:33,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30547.03 MB 2025-02-14 23:27:33,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22625.49 MB 2025-02-14 23:27:33,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7921.53 MB 2025-02-14 23:27:33,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35777.41 MB 2025-02-14 23:27:33,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35777.41 MB 2025-02-14 23:27:33,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:27:33,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33045.18 MB 2025-02-14 23:27:33,548 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 23:27:33,548 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:27:33,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:27:33,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:27:33,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:27:33,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:27:33,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22625.49 MB 2025-02-14 23:27:33,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31018.77 MB 2025-02-14 23:27:33,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 23:27:33,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35777.41 MB 2025-02-14 23:27:33,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44124.08 MB 2025-02-14 23:27:33,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 23:27:33,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31018.77 MB 2025-02-14 23:27:33,715 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 23:27:33,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:27:33,716 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:27:33,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:27:33,717 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:27:33,722 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:27:33,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:27:33,723 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:27:33,723 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:28:25,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:28:25,428 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:28:25,433 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:28:25,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:28:25,437 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1097, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:28:25,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:28:25,438 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1097, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:28:42,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:28:42,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:28:42,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.93 seconds 2025-02-14 23:28:42,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:42,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20612.78 MB 2025-02-14 23:28:42,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24495.00 MB 2025-02-14 23:28:42,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3882.22 MB 2025-02-14 23:28:42,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52470.74 MB 2025-02-14 23:28:42,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26180.85 MB 2025-02-14 23:28:42,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26289.90 MB 2025-02-14 23:28:42,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33482.34 MB 2025-02-14 23:28:42,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:28:42,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:28:42,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:28:42,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:42,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24495.00 MB 2025-02-14 23:28:42,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21481.86 MB 2025-02-14 23:28:42,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3013.14 MB 2025-02-14 23:28:42,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26180.85 MB 2025-02-14 23:28:42,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44291.85 MB 2025-02-14 23:28:42,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18111.00 MB 2025-02-14 23:28:42,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35994.15 MB 2025-02-14 23:28:44,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:28:44,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:28:44,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 23:28:44,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:44,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21481.86 MB 2025-02-14 23:28:44,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22012.70 MB 2025-02-14 23:28:44,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:28:44,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44291.85 MB 2025-02-14 23:28:44,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25130.17 MB 2025-02-14 23:28:44,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19161.68 MB 2025-02-14 23:28:44,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25992.29 MB 2025-02-14 23:28:44,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:28:44,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:28:44,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:28:44,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:44,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22012.70 MB 2025-02-14 23:28:44,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23902.24 MB 2025-02-14 23:28:44,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:28:44,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25130.17 MB 2025-02-14 23:28:44,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27017.61 MB 2025-02-14 23:28:44,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 23:28:44,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25319.66 MB 2025-02-14 23:28:44,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:28:44,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:28:44,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:28:44,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:44,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23902.24 MB 2025-02-14 23:28:44,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26144.09 MB 2025-02-14 23:28:44,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:28:44,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27017.61 MB 2025-02-14 23:28:44,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33151.78 MB 2025-02-14 23:28:44,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:28:44,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31688.37 MB 2025-02-14 23:28:44,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:28:44,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:28:44,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 23:28:44,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:44,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22012.70 MB 2025-02-14 23:28:44,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26144.09 MB 2025-02-14 23:28:44,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:28:44,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25130.17 MB 2025-02-14 23:28:44,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33151.78 MB 2025-02-14 23:28:44,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 23:28:44,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31688.37 MB 2025-02-14 23:28:44,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:28:44,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:28:44,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:28:44,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:44,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27677.63 MB 2025-02-14 23:28:44,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28444.64 MB 2025-02-14 23:28:44,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:28:44,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33151.78 MB 2025-02-14 23:28:44,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33567.01 MB 2025-02-14 23:28:44,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:28:44,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29152.42 MB 2025-02-14 23:28:44,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:28:44,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:28:44,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:28:44,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:44,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28857.52 MB 2025-02-14 23:28:44,821 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29086.69 MB 2025-02-14 23:28:44,821 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-14 23:28:44,821 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33567.01 MB 2025-02-14 23:28:44,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33567.01 MB 2025-02-14 23:28:44,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:28:44,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29297.73 MB 2025-02-14 23:28:44,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:28:44,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:28:44,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.38 seconds 2025-02-14 23:28:44,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:44,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16790.74 MB 2025-02-14 23:28:44,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29287.77 MB 2025-02-14 23:28:44,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12497.03 MB 2025-02-14 23:28:44,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52470.74 MB 2025-02-14 23:28:44,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33567.01 MB 2025-02-14 23:28:44,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18903.73 MB 2025-02-14 23:28:44,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29297.73 MB 2025-02-14 23:28:45,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:28:45,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:28:45,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:28:45,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:45,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29287.77 MB 2025-02-14 23:28:45,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21795.13 MB 2025-02-14 23:28:45,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7492.64 MB 2025-02-14 23:28:45,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33567.01 MB 2025-02-14 23:28:45,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33567.01 MB 2025-02-14 23:28:45,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:28:45,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31799.43 MB 2025-02-14 23:28:45,111 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:28:45,111 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:28:45,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:28:45,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:28:45,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:28:45,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:28:45,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21795.13 MB 2025-02-14 23:28:45,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30234.15 MB 2025-02-14 23:28:45,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:28:45,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33567.01 MB 2025-02-14 23:28:45,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41957.72 MB 2025-02-14 23:28:45,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:28:45,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30234.15 MB 2025-02-14 23:28:45,274 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:28:45,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:28:45,275 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:28:45,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:28:45,276 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:28:45,281 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:28:45,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:28:45,282 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:28:45,282 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:30:08,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:30:08,663 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:30:08,668 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:30:08,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:30:08,672 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:30:08,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:30:08,673 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:30:26,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:30:26,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:30:26,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.19 seconds 2025-02-14 23:30:26,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:26,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-14 23:30:26,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-14 23:30:26,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-14 23:30:26,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54542.73 MB 2025-02-14 23:30:26,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30679.24 MB 2025-02-14 23:30:26,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23863.49 MB 2025-02-14 23:30:26,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34307.29 MB 2025-02-14 23:30:26,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:30:26,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:30:26,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:30:26,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:26,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-14 23:30:26,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21927.90 MB 2025-02-14 23:30:26,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3470.71 MB 2025-02-14 23:30:26,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30679.24 MB 2025-02-14 23:30:26,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43775.95 MB 2025-02-14 23:30:26,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13096.71 MB 2025-02-14 23:30:26,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37984.72 MB 2025-02-14 23:30:28,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:30:28,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:30:28,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:30:28,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:28,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21927.90 MB 2025-02-14 23:30:28,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.74 MB 2025-02-14 23:30:28,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:30:28,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43775.95 MB 2025-02-14 23:30:28,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27906.80 MB 2025-02-14 23:30:28,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15869.15 MB 2025-02-14 23:30:28,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26437.29 MB 2025-02-14 23:30:28,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:30:28,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:30:28,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:30:28,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:28,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 23:30:28,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24348.27 MB 2025-02-14 23:30:28,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:30:28,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 23:30:28,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27906.80 MB 2025-02-14 23:30:28,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:30:28,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25765.70 MB 2025-02-14 23:30:29,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:30:29,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:30:29,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:30:29,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:29,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24348.27 MB 2025-02-14 23:30:29,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 23:30:29,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:30:29,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 23:30:29,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34040.97 MB 2025-02-14 23:30:29,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:30:29,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 23:30:29,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:30:29,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:30:29,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:30:29,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:29,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 23:30:29,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 23:30:29,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:30:29,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27906.80 MB 2025-02-14 23:30:29,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34040.97 MB 2025-02-14 23:30:29,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:30:29,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 23:30:29,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:30:29,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:30:29,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:30:29,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:29,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28123.67 MB 2025-02-14 23:30:29,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28890.67 MB 2025-02-14 23:30:29,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:30:29,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34040.97 MB 2025-02-14 23:30:29,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34456.21 MB 2025-02-14 23:30:29,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:30:29,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.46 MB 2025-02-14 23:30:29,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:30:29,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:30:29,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:30:29,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:29,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29303.56 MB 2025-02-14 23:30:29,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29530.79 MB 2025-02-14 23:30:29,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.23 MB 2025-02-14 23:30:29,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34456.21 MB 2025-02-14 23:30:29,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34456.21 MB 2025-02-14 23:30:29,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:30:29,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29774.60 MB 2025-02-14 23:30:29,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:30:29,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:30:29,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.60 seconds 2025-02-14 23:30:29,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:29,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-14 23:30:29,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29731.00 MB 2025-02-14 23:30:29,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12640.63 MB 2025-02-14 23:30:29,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54542.73 MB 2025-02-14 23:30:29,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34456.21 MB 2025-02-14 23:30:29,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20086.52 MB 2025-02-14 23:30:29,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29774.60 MB 2025-02-14 23:30:29,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:30:29,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:30:29,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:30:29,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:29,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29731.00 MB 2025-02-14 23:30:29,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22081.70 MB 2025-02-14 23:30:29,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7649.30 MB 2025-02-14 23:30:29,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34456.21 MB 2025-02-14 23:30:29,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34456.21 MB 2025-02-14 23:30:29,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:30:29,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32232.19 MB 2025-02-14 23:30:29,562 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 23:30:29,563 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:30:29,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:30:29,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:30:29,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:30:29,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:30:29,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22081.70 MB 2025-02-14 23:30:29,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30485.26 MB 2025-02-14 23:30:29,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 23:30:29,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34456.21 MB 2025-02-14 23:30:29,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42811.26 MB 2025-02-14 23:30:29,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 23:30:29,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30485.26 MB 2025-02-14 23:30:29,725 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 23:30:29,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:30:29,727 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:30:29,728 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:30:29,728 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:30:29,732 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:30:29,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:30:29,734 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:30:29,734 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:31:47,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:31:47,823 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:31:47,828 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:31:47,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:31:47,832 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1790, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:31:47,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:31:47,833 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1790, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:32:15,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:32:15,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:32:15,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.54 seconds 2025-02-14 23:32:15,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:15,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25441.71 MB 2025-02-14 23:32:15,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31777.21 MB 2025-02-14 23:32:15,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6335.50 MB 2025-02-14 23:32:15,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51166.31 MB 2025-02-14 23:32:15,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36983.28 MB 2025-02-14 23:32:15,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14183.04 MB 2025-02-14 23:32:15,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40576.20 MB 2025-02-14 23:32:15,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:32:15,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:32:15,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 23:32:15,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:15,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31777.21 MB 2025-02-14 23:32:15,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25083.50 MB 2025-02-14 23:32:15,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6693.71 MB 2025-02-14 23:32:15,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36983.28 MB 2025-02-14 23:32:15,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59257.13 MB 2025-02-14 23:32:15,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22273.85 MB 2025-02-14 23:32:15,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50093.38 MB 2025-02-14 23:32:17,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:32:17,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:32:17,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:32:17,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:17,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25083.50 MB 2025-02-14 23:32:17,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25614.34 MB 2025-02-14 23:32:17,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:32:17,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59257.13 MB 2025-02-14 23:32:17,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32063.36 MB 2025-02-14 23:32:17,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27193.77 MB 2025-02-14 23:32:17,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29592.89 MB 2025-02-14 23:32:17,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:32:17,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:32:17,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:32:17,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:17,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25614.34 MB 2025-02-14 23:32:17,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27503.88 MB 2025-02-14 23:32:17,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:32:17,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32063.36 MB 2025-02-14 23:32:17,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32063.36 MB 2025-02-14 23:32:17,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:32:17,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28921.31 MB 2025-02-14 23:32:17,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:32:17,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:32:17,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:32:17,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:17,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27503.88 MB 2025-02-14 23:32:17,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29745.73 MB 2025-02-14 23:32:17,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:32:17,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32063.36 MB 2025-02-14 23:32:17,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37253.81 MB 2025-02-14 23:32:17,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 23:32:17,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35290.02 MB 2025-02-14 23:32:17,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:32:17,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:32:17,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:32:17,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:17,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25614.34 MB 2025-02-14 23:32:17,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29745.73 MB 2025-02-14 23:32:17,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:32:17,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32063.36 MB 2025-02-14 23:32:17,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37253.81 MB 2025-02-14 23:32:17,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 23:32:17,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35290.02 MB 2025-02-14 23:32:17,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:32:17,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:32:17,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:32:17,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:17,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31279.28 MB 2025-02-14 23:32:17,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32046.28 MB 2025-02-14 23:32:17,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:32:17,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37253.81 MB 2025-02-14 23:32:17,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37669.04 MB 2025-02-14 23:32:17,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:32:17,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32754.07 MB 2025-02-14 23:32:17,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:32:17,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:32:17,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:32:17,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:17,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32459.17 MB 2025-02-14 23:32:17,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32688.51 MB 2025-02-14 23:32:17,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.34 MB 2025-02-14 23:32:17,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37669.04 MB 2025-02-14 23:32:17,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37669.04 MB 2025-02-14 23:32:17,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:32:17,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32900.52 MB 2025-02-14 23:32:17,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:32:17,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:32:17,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.03 seconds 2025-02-14 23:32:17,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:17,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19205.21 MB 2025-02-14 23:32:17,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32889.24 MB 2025-02-14 23:32:17,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13684.03 MB 2025-02-14 23:32:17,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51166.31 MB 2025-02-14 23:32:17,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37669.04 MB 2025-02-14 23:32:17,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13497.27 MB 2025-02-14 23:32:17,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32900.52 MB 2025-02-14 23:32:18,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:32:18,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:32:18,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:32:18,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:18,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32889.24 MB 2025-02-14 23:32:18,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24204.27 MB 2025-02-14 23:32:18,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8684.97 MB 2025-02-14 23:32:18,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37669.04 MB 2025-02-14 23:32:18,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37669.04 MB 2025-02-14 23:32:18,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:32:18,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35396.60 MB 2025-02-14 23:32:18,148 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 23:32:18,148 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 23:32:18,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:32:18,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:32:18,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:32:18,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:18,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24204.27 MB 2025-02-14 23:32:18,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32629.22 MB 2025-02-14 23:32:18,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 23:32:18,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37669.04 MB 2025-02-14 23:32:18,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46045.07 MB 2025-02-14 23:32:18,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 23:32:18,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32629.22 MB 2025-02-14 23:32:18,311 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 23:32:18,312 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:32:18,312 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:32:18,313 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:32:18,313 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:32:18,318 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:32:18,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:32:18,319 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:32:18,319 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 23:32:29,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:32:29,718 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:32:29,723 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:32:29,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:32:29,726 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1754, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:32:29,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:32:29,727 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1754, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:32:57,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:32:57,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:32:57,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.29 seconds 2025-02-14 23:32:57,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:57,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25190.86 MB 2025-02-14 23:32:57,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31398.43 MB 2025-02-14 23:32:57,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6207.57 MB 2025-02-14 23:32:57,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54421.09 MB 2025-02-14 23:32:57,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36855.35 MB 2025-02-14 23:32:57,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17565.75 MB 2025-02-14 23:32:57,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40325.35 MB 2025-02-14 23:32:57,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:32:57,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:32:57,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 23:32:57,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:57,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31398.43 MB 2025-02-14 23:32:57,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24896.35 MB 2025-02-14 23:32:57,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6502.08 MB 2025-02-14 23:32:57,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36855.35 MB 2025-02-14 23:32:57,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59118.71 MB 2025-02-14 23:32:57,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22263.37 MB 2025-02-14 23:32:57,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49845.15 MB 2025-02-14 23:32:59,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:32:59,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:32:59,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 23:32:59,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24896.35 MB 2025-02-14 23:32:59,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25427.19 MB 2025-02-14 23:32:59,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:32:59,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59118.71 MB 2025-02-14 23:32:59,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27885.83 MB 2025-02-14 23:32:59,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31232.88 MB 2025-02-14 23:32:59,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29406.78 MB 2025-02-14 23:32:59,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:32:59,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:32:59,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:32:59,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-14 23:32:59,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27316.72 MB 2025-02-14 23:32:59,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:32:59,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27885.83 MB 2025-02-14 23:32:59,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30716.99 MB 2025-02-14 23:32:59,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 23:32:59,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28734.15 MB 2025-02-14 23:32:59,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:32:59,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:32:59,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 23:32:59,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27316.72 MB 2025-02-14 23:32:59,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-14 23:32:59,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:32:59,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30716.99 MB 2025-02-14 23:32:59,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36851.15 MB 2025-02-14 23:32:59,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:32:59,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-14 23:32:59,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:32:59,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:32:59,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:32:59,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-14 23:32:59,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-14 23:32:59,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:32:59,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27885.83 MB 2025-02-14 23:32:59,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36851.15 MB 2025-02-14 23:32:59,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 23:32:59,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-14 23:32:59,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:32:59,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:32:59,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:32:59,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31092.12 MB 2025-02-14 23:32:59,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31859.12 MB 2025-02-14 23:32:59,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:32:59,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36851.15 MB 2025-02-14 23:32:59,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37266.39 MB 2025-02-14 23:32:59,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:32:59,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32566.91 MB 2025-02-14 23:32:59,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:32:59,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:32:59,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:32:59,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32272.01 MB 2025-02-14 23:32:59,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32499.77 MB 2025-02-14 23:32:59,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.76 MB 2025-02-14 23:32:59,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37266.39 MB 2025-02-14 23:32:59,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37266.39 MB 2025-02-14 23:32:59,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:32:59,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32736.09 MB 2025-02-14 23:32:59,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:32:59,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:32:59,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.84 seconds 2025-02-14 23:32:59,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19079.78 MB 2025-02-14 23:32:59,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32700.01 MB 2025-02-14 23:32:59,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13620.23 MB 2025-02-14 23:32:59,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54421.09 MB 2025-02-14 23:32:59,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37266.39 MB 2025-02-14 23:32:59,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17154.70 MB 2025-02-14 23:32:59,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32736.09 MB 2025-02-14 23:32:59,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:32:59,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:32:59,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:32:59,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32700.01 MB 2025-02-14 23:32:59,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24071.47 MB 2025-02-14 23:32:59,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8628.54 MB 2025-02-14 23:32:59,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37266.39 MB 2025-02-14 23:32:59,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37266.39 MB 2025-02-14 23:32:59,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:32:59,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35201.23 MB 2025-02-14 23:32:59,859 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 23:32:59,859 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:32:59,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:32:59,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:32:59,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:32:59,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:32:59,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24071.47 MB 2025-02-14 23:32:59,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32476.55 MB 2025-02-14 23:32:59,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-14 23:32:59,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37266.39 MB 2025-02-14 23:32:59,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45621.44 MB 2025-02-14 23:32:59,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 23:32:59,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32476.55 MB 2025-02-14 23:33:00,022 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 23:33:00,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:00,023 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:33:00,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:00,024 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:33:00,029 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:33:00,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:00,030 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:33:00,030 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:33:45,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:45,058 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:33:45,065 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:33:45,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:45,071 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 212, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:33:45,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:45,073 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 212, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:33:48,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:33:48,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:33:48,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.37 seconds 2025-02-14 23:33:48,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:48,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.96 MB 2025-02-14 23:33:48,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15196.21 MB 2025-02-14 23:33:48,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.26 MB 2025-02-14 23:33:48,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53976.50 MB 2025-02-14 23:33:48,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20224.93 MB 2025-02-14 23:33:48,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33751.56 MB 2025-02-14 23:33:48,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24143.82 MB 2025-02-14 23:33:48,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:33:48,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:33:48,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:33:48,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:48,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15196.21 MB 2025-02-14 23:33:48,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15440.32 MB 2025-02-14 23:33:48,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 244.11 MB 2025-02-14 23:33:48,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20224.93 MB 2025-02-14 23:33:48,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20224.93 MB 2025-02-14 23:33:48,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:33:48,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17950.29 MB 2025-02-14 23:33:49,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:33:49,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:33:49,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-14 23:33:49,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15440.32 MB 2025-02-14 23:33:49,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15699.10 MB 2025-02-14 23:33:49,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 258.79 MB 2025-02-14 23:33:49,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20224.93 MB 2025-02-14 23:33:49,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20224.93 MB 2025-02-14 23:33:49,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:33:49,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19695.77 MB 2025-02-14 23:33:49,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:33:49,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:33:49,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:33:49,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15699.04 MB 2025-02-14 23:33:49,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16619.96 MB 2025-02-14 23:33:49,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 920.92 MB 2025-02-14 23:33:49,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20224.93 MB 2025-02-14 23:33:49,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20224.93 MB 2025-02-14 23:33:49,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:33:49,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17310.96 MB 2025-02-14 23:33:49,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:33:49,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:33:49,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 23:33:49,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16619.96 MB 2025-02-14 23:33:49,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17712.90 MB 2025-02-14 23:33:49,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1092.94 MB 2025-02-14 23:33:49,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20224.93 MB 2025-02-14 23:33:49,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22070.43 MB 2025-02-14 23:33:49,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1845.49 MB 2025-02-14 23:33:49,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20415.70 MB 2025-02-14 23:33:49,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:33:49,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:33:49,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 23:33:49,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15699.04 MB 2025-02-14 23:33:49,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17712.90 MB 2025-02-14 23:33:49,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2013.86 MB 2025-02-14 23:33:49,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20224.93 MB 2025-02-14 23:33:49,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22070.43 MB 2025-02-14 23:33:49,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1845.49 MB 2025-02-14 23:33:49,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20415.70 MB 2025-02-14 23:33:49,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:33:49,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:33:49,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:33:49,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18460.50 MB 2025-02-14 23:33:49,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.42 MB 2025-02-14 23:33:49,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 373.91 MB 2025-02-14 23:33:49,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22070.43 MB 2025-02-14 23:33:49,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22271.75 MB 2025-02-14 23:33:49,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-14 23:33:49,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19182.47 MB 2025-02-14 23:33:49,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:33:49,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:33:49,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:33:49,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19035.71 MB 2025-02-14 23:33:49,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19244.77 MB 2025-02-14 23:33:49,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.07 MB 2025-02-14 23:33:49,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22271.75 MB 2025-02-14 23:33:49,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22271.75 MB 2025-02-14 23:33:49,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:33:49,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19286.21 MB 2025-02-14 23:33:49,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:33:49,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:33:49,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.54 seconds 2025-02-14 23:33:49,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-14 23:33:49,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19445.30 MB 2025-02-14 23:33:49,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5737.97 MB 2025-02-14 23:33:49,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53976.50 MB 2025-02-14 23:33:49,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22271.75 MB 2025-02-14 23:33:49,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31704.74 MB 2025-02-14 23:33:49,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19445.30 MB 2025-02-14 23:33:49,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:33:49,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:33:49,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:33:49,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14729.96 MB 2025-02-14 23:33:49,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17735.88 MB 2025-02-14 23:33:49,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.92 MB 2025-02-14 23:33:49,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22271.75 MB 2025-02-14 23:33:49,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22271.75 MB 2025-02-14 23:33:49,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:33:49,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18036.44 MB 2025-02-14 23:33:49,901 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 23:33:49,901 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:33:49,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:33:49,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:33:49,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:33:49,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:33:49,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17735.88 MB 2025-02-14 23:33:49,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26152.49 MB 2025-02-14 23:33:49,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 23:33:49,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22271.75 MB 2025-02-14 23:33:49,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30639.39 MB 2025-02-14 23:33:49,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 23:33:49,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26152.49 MB 2025-02-14 23:33:50,069 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 23:33:50,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:50,071 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:33:50,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:50,072 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:33:50,076 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:33:50,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:33:50,077 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:33:50,078 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:35:19,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:35:19,002 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:35:19,010 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:35:19,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:35:19,017 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 898, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:35:19,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:35:19,019 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 898, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:35:32,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:35:32,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:35:32,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.76 seconds 2025-02-14 23:35:32,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:32,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19226.11 MB 2025-02-14 23:35:32,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22404.09 MB 2025-02-14 23:35:32,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3177.97 MB 2025-02-14 23:35:32,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39007.03 MB 2025-02-14 23:35:32,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27116.18 MB 2025-02-14 23:35:32,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11890.85 MB 2025-02-14 23:35:32,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31415.39 MB 2025-02-14 23:35:32,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:35:32,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:35:32,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 23:35:32,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:32,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22404.09 MB 2025-02-14 23:35:32,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20447.32 MB 2025-02-14 23:35:32,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1956.76 MB 2025-02-14 23:35:32,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27116.18 MB 2025-02-14 23:35:32,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36251.37 MB 2025-02-14 23:35:32,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9135.19 MB 2025-02-14 23:35:32,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32347.54 MB 2025-02-14 23:35:34,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:35:34,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:35:34,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 23:35:34,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:34,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20447.32 MB 2025-02-14 23:35:34,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20978.16 MB 2025-02-14 23:35:34,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:35:34,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36251.37 MB 2025-02-14 23:35:34,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26061.31 MB 2025-02-14 23:35:34,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10190.06 MB 2025-02-14 23:35:34,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24956.71 MB 2025-02-14 23:35:34,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:35:34,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:35:34,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:35:34,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:34,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20978.16 MB 2025-02-14 23:35:34,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22867.70 MB 2025-02-14 23:35:34,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:35:34,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26061.31 MB 2025-02-14 23:35:34,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26061.31 MB 2025-02-14 23:35:34,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:35:34,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24285.13 MB 2025-02-14 23:35:34,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:35:34,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:35:34,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:35:34,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:34,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22867.70 MB 2025-02-14 23:35:34,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25109.55 MB 2025-02-14 23:35:34,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:35:34,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26061.31 MB 2025-02-14 23:35:34,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32667.34 MB 2025-02-14 23:35:34,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:35:34,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30653.83 MB 2025-02-14 23:35:34,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:35:34,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:35:34,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:35:34,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:34,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20978.16 MB 2025-02-14 23:35:34,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25109.55 MB 2025-02-14 23:35:34,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:35:34,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26061.31 MB 2025-02-14 23:35:34,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32667.34 MB 2025-02-14 23:35:34,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:35:34,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30653.83 MB 2025-02-14 23:35:35,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:35:35,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:35:35,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:35:35,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:35,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26643.10 MB 2025-02-14 23:35:35,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27410.10 MB 2025-02-14 23:35:35,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:35:35,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32667.34 MB 2025-02-14 23:35:35,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 23:35:35,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:35:35,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28117.89 MB 2025-02-14 23:35:35,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:35:35,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:35:35,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:35:35,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:35,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27822.99 MB 2025-02-14 23:35:35,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28051.95 MB 2025-02-14 23:35:35,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 23:35:35,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 23:35:35,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 23:35:35,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:35:35,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28250.52 MB 2025-02-14 23:35:35,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:35:35,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:35:35,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.15 seconds 2025-02-14 23:35:35,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:35,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16097.41 MB 2025-02-14 23:35:35,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28252.82 MB 2025-02-14 23:35:35,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12155.41 MB 2025-02-14 23:35:35,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39007.03 MB 2025-02-14 23:35:35,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 23:35:35,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5924.45 MB 2025-02-14 23:35:35,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28252.82 MB 2025-02-14 23:35:35,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:35:35,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:35:35,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:35:35,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:35,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28252.82 MB 2025-02-14 23:35:35,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21098.75 MB 2025-02-14 23:35:35,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7154.07 MB 2025-02-14 23:35:35,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 23:35:35,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33082.57 MB 2025-02-14 23:35:35,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:35:35,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30762.03 MB 2025-02-14 23:35:35,454 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 23:35:35,455 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:35:35,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:35:35,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:35:35,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:35:35,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:35:35,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21098.75 MB 2025-02-14 23:35:35,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29529.43 MB 2025-02-14 23:35:35,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 23:35:35,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33082.57 MB 2025-02-14 23:35:35,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41464.89 MB 2025-02-14 23:35:35,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 23:35:35,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29529.43 MB 2025-02-14 23:35:35,620 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 23:35:35,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:35:35,621 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:35:35,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:35:35,622 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:35:35,627 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:35:35,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:35:35,628 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:35:35,628 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:36:28,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:36:28,640 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:36:28,645 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:36:28,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:36:28,649 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:36:28,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:36:28,650 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:36:59,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:36:59,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:36:59,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.90 seconds 2025-02-14 23:36:59,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:36:59,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26932.90 MB 2025-02-14 23:36:59,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34025.47 MB 2025-02-14 23:36:59,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7092.57 MB 2025-02-14 23:36:59,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54037.32 MB 2025-02-14 23:36:59,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37769.71 MB 2025-02-14 23:36:59,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16267.61 MB 2025-02-14 23:36:59,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42973.36 MB 2025-02-14 23:36:59,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:36:59,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:36:59,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 23:36:59,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:36:59,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34025.47 MB 2025-02-14 23:36:59,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26197.07 MB 2025-02-14 23:36:59,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7828.40 MB 2025-02-14 23:36:59,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37769.71 MB 2025-02-14 23:36:59,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65309.51 MB 2025-02-14 23:36:59,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 27539.80 MB 2025-02-14 23:36:59,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54563.08 MB 2025-02-14 23:37:01,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:37:01,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:37:01,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 23:37:01,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:01,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26197.07 MB 2025-02-14 23:37:01,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26727.91 MB 2025-02-14 23:37:01,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:37:01,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65309.51 MB 2025-02-14 23:37:01,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32801.55 MB 2025-02-14 23:37:01,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32507.95 MB 2025-02-14 23:37:01,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30706.46 MB 2025-02-14 23:37:01,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:37:01,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:37:01,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:37:01,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:01,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26727.91 MB 2025-02-14 23:37:01,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28617.45 MB 2025-02-14 23:37:01,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:37:01,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32801.55 MB 2025-02-14 23:37:01,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32801.55 MB 2025-02-14 23:37:01,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:37:01,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30034.87 MB 2025-02-14 23:37:01,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:37:01,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:37:01,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:37:01,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:01,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28617.45 MB 2025-02-14 23:37:01,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30859.30 MB 2025-02-14 23:37:01,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:37:01,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32801.55 MB 2025-02-14 23:37:01,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38463.86 MB 2025-02-14 23:37:01,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:37:01,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36403.58 MB 2025-02-14 23:37:01,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:37:01,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:37:01,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:37:01,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:01,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26727.91 MB 2025-02-14 23:37:01,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30859.30 MB 2025-02-14 23:37:01,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:37:01,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32801.55 MB 2025-02-14 23:37:01,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38463.86 MB 2025-02-14 23:37:01,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:37:01,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36403.58 MB 2025-02-14 23:37:02,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:37:02,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:37:02,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:37:02,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:02,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32392.84 MB 2025-02-14 23:37:02,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33159.85 MB 2025-02-14 23:37:02,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:37:02,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38463.86 MB 2025-02-14 23:37:02,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38881.20 MB 2025-02-14 23:37:02,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:37:02,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33867.63 MB 2025-02-14 23:37:02,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:37:02,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:37:02,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:37:02,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:02,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33572.73 MB 2025-02-14 23:37:02,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33800.71 MB 2025-02-14 23:37:02,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 23:37:02,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38881.20 MB 2025-02-14 23:37:02,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38881.20 MB 2025-02-14 23:37:02,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:37:02,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34011.61 MB 2025-02-14 23:37:02,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:37:02,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:37:02,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.39 seconds 2025-02-14 23:37:02,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:02,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-14 23:37:02,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34001.19 MB 2025-02-14 23:37:02,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14050.39 MB 2025-02-14 23:37:02,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54037.32 MB 2025-02-14 23:37:02,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38881.20 MB 2025-02-14 23:37:02,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15156.12 MB 2025-02-14 23:37:02,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34011.61 MB 2025-02-14 23:37:02,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:37:02,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:37:02,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 23:37:02,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:02,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34001.19 MB 2025-02-14 23:37:02,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24937.50 MB 2025-02-14 23:37:02,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9063.70 MB 2025-02-14 23:37:02,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38881.20 MB 2025-02-14 23:37:02,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38881.20 MB 2025-02-14 23:37:02,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:37:02,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36498.12 MB 2025-02-14 23:37:02,327 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 23:37:02,328 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:37:02,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:37:02,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:37:02,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:37:02,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:02,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24937.50 MB 2025-02-14 23:37:02,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33326.64 MB 2025-02-14 23:37:02,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 23:37:02,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38881.20 MB 2025-02-14 23:37:02,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43052.43 MB 2025-02-14 23:37:02,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 23:37:02,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33326.64 MB 2025-02-14 23:37:02,489 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 23:37:02,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:02,491 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:37:02,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:02,492 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:37:02,496 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:37:02,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:02,497 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:37:02,497 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:37:25,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:25,871 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:37:25,875 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:37:25,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:25,879 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1297, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:37:25,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:25,880 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1297, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:37:45,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:37:45,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:37:45,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.11 seconds 2025-02-14 23:37:45,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:45,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22006.41 MB 2025-02-14 23:37:45,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26597.08 MB 2025-02-14 23:37:45,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4590.67 MB 2025-02-14 23:37:45,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51394.90 MB 2025-02-14 23:37:45,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35219.57 MB 2025-02-14 23:37:45,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16175.33 MB 2025-02-14 23:37:45,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35554.64 MB 2025-02-14 23:37:46,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:37:46,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:37:46,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 23:37:46,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:46,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26597.08 MB 2025-02-14 23:37:46,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22520.55 MB 2025-02-14 23:37:46,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4076.53 MB 2025-02-14 23:37:46,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35219.57 MB 2025-02-14 23:37:46,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41072.72 MB 2025-02-14 23:37:46,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5853.15 MB 2025-02-14 23:37:46,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36752.83 MB 2025-02-14 23:37:47,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:37:47,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:37:47,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:37:47,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:47,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22520.55 MB 2025-02-14 23:37:47,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23051.39 MB 2025-02-14 23:37:47,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:37:47,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41072.72 MB 2025-02-14 23:37:47,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27873.25 MB 2025-02-14 23:37:47,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13199.47 MB 2025-02-14 23:37:47,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27029.94 MB 2025-02-14 23:37:47,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:37:47,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:37:47,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:37:47,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:47,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23051.39 MB 2025-02-14 23:37:47,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24940.92 MB 2025-02-14 23:37:47,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:37:47,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 23:37:47,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27873.25 MB 2025-02-14 23:37:47,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:37:47,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26358.35 MB 2025-02-14 23:37:48,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:37:48,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:37:48,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:37:48,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:48,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24940.92 MB 2025-02-14 23:37:48,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27182.78 MB 2025-02-14 23:37:48,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:37:48,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 23:37:48,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34479.28 MB 2025-02-14 23:37:48,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:37:48,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32727.06 MB 2025-02-14 23:37:48,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:37:48,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:37:48,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 23:37:48,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:48,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23051.39 MB 2025-02-14 23:37:48,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27182.78 MB 2025-02-14 23:37:48,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:37:48,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27873.25 MB 2025-02-14 23:37:48,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34479.28 MB 2025-02-14 23:37:48,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:37:48,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32727.06 MB 2025-02-14 23:37:48,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:37:48,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:37:48,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:37:48,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:48,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28716.32 MB 2025-02-14 23:37:48,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29483.32 MB 2025-02-14 23:37:48,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:37:48,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34479.28 MB 2025-02-14 23:37:48,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 23:37:48,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:37:48,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30191.11 MB 2025-02-14 23:37:48,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:37:48,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:37:48,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:37:48,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:48,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29896.21 MB 2025-02-14 23:37:48,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30126.27 MB 2025-02-14 23:37:48,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.05 MB 2025-02-14 23:37:48,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34894.51 MB 2025-02-14 23:37:48,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 23:37:48,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:37:48,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30348.20 MB 2025-02-14 23:37:48,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:37:48,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:37:48,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.52 seconds 2025-02-14 23:37:48,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:48,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17487.56 MB 2025-02-14 23:37:48,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30327.34 MB 2025-02-14 23:37:48,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12839.78 MB 2025-02-14 23:37:48,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51394.90 MB 2025-02-14 23:37:48,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 23:37:48,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16500.39 MB 2025-02-14 23:37:48,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30348.20 MB 2025-02-14 23:37:48,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:37:48,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:37:48,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:37:48,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:48,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30327.34 MB 2025-02-14 23:37:48,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22491.95 MB 2025-02-14 23:37:48,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7835.39 MB 2025-02-14 23:37:48,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34894.51 MB 2025-02-14 23:37:48,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34894.51 MB 2025-02-14 23:37:48,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:37:48,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32839.01 MB 2025-02-14 23:37:48,684 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:37:48,684 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:37:48,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:37:48,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:37:48,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:37:48,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:37:48,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22491.95 MB 2025-02-14 23:37:48,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30930.97 MB 2025-02-14 23:37:48,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:37:48,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34894.51 MB 2025-02-14 23:37:48,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43285.22 MB 2025-02-14 23:37:48,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:37:48,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30930.97 MB 2025-02-14 23:37:48,847 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:37:48,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:48,848 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:37:48,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:48,849 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:37:48,854 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:37:48,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:37:48,855 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:37:48,855 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:38:28,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:38:28,341 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:38:28,349 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:38:28,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:38:28,356 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 492, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:38:28,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:38:28,358 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 492, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:38:36,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:38:36,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:38:36,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.67 seconds 2025-02-14 23:38:36,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:36,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16397.04 MB 2025-02-14 23:38:36,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18138.20 MB 2025-02-14 23:38:36,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1741.16 MB 2025-02-14 23:38:36,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55870.23 MB 2025-02-14 23:38:36,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25167.92 MB 2025-02-14 23:38:36,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30702.31 MB 2025-02-14 23:38:36,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27000.87 MB 2025-02-14 23:38:36,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:38:36,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:38:36,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 23:38:36,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:36,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18138.20 MB 2025-02-14 23:38:36,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18336.65 MB 2025-02-14 23:38:36,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 198.45 MB 2025-02-14 23:38:36,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25167.92 MB 2025-02-14 23:38:36,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29395.78 MB 2025-02-14 23:38:36,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4227.86 MB 2025-02-14 23:38:36,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25700.30 MB 2025-02-14 23:38:37,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:38:37,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:38:37,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:38:37,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:37,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18336.65 MB 2025-02-14 23:38:37,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18867.50 MB 2025-02-14 23:38:37,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:38:37,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29395.78 MB 2025-02-14 23:38:37,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26348.62 MB 2025-02-14 23:38:37,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3047.16 MB 2025-02-14 23:38:37,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22846.04 MB 2025-02-14 23:38:38,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:38:38,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:38:38,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:38:38,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18867.50 MB 2025-02-14 23:38:38,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20757.03 MB 2025-02-14 23:38:38,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:38:38,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26348.62 MB 2025-02-14 23:38:38,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26348.62 MB 2025-02-14 23:38:38,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:38:38,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22174.46 MB 2025-02-14 23:38:38,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:38:38,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:38:38,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:38:38,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20757.03 MB 2025-02-14 23:38:38,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22998.89 MB 2025-02-14 23:38:38,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:38:38,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26348.62 MB 2025-02-14 23:38:38,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30832.33 MB 2025-02-14 23:38:38,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4483.71 MB 2025-02-14 23:38:38,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28543.17 MB 2025-02-14 23:38:38,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:38:38,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:38:38,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 23:38:38,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18867.50 MB 2025-02-14 23:38:38,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22998.89 MB 2025-02-14 23:38:38,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:38:38,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26348.62 MB 2025-02-14 23:38:38,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30832.33 MB 2025-02-14 23:38:38,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4483.71 MB 2025-02-14 23:38:38,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28543.17 MB 2025-02-14 23:38:38,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:38:38,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:38:38,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:38:38,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24532.43 MB 2025-02-14 23:38:38,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25299.43 MB 2025-02-14 23:38:38,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:38:38,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30832.33 MB 2025-02-14 23:38:38,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 23:38:38,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 23:38:38,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26007.22 MB 2025-02-14 23:38:38,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:38:38,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:38:38,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:38:38,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25712.32 MB 2025-02-14 23:38:38,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25940.72 MB 2025-02-14 23:38:38,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-14 23:38:38,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 23:38:38,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 23:38:38,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:38:38,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26130.43 MB 2025-02-14 23:38:38,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:38:38,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:38:38,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.06 seconds 2025-02-14 23:38:38,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14682.87 MB 2025-02-14 23:38:38,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26141.79 MB 2025-02-14 23:38:38,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11458.91 MB 2025-02-14 23:38:38,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55870.23 MB 2025-02-14 23:38:38,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 23:38:38,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24624.76 MB 2025-02-14 23:38:38,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26141.79 MB 2025-02-14 23:38:38,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:38:38,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:38:38,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:38:38,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26141.79 MB 2025-02-14 23:38:38,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19687.26 MB 2025-02-14 23:38:38,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6454.52 MB 2025-02-14 23:38:38,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 23:38:38,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 23:38:38,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:38:38,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28653.45 MB 2025-02-14 23:38:38,711 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:38:38,711 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 23:38:38,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:38:38,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:38:38,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:38:38,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:38:38,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.26 MB 2025-02-14 23:38:38,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28126.29 MB 2025-02-14 23:38:38,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:38:38,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 23:38:38,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39636.17 MB 2025-02-14 23:38:38,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:38:38,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28126.29 MB 2025-02-14 23:38:38,887 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:38:38,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:38:38,889 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:38:38,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:38:38,890 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:38:38,894 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:38:38,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:38:38,896 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:38:38,896 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 23:39:05,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:05,191 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:39:05,196 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:39:05,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:05,199 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 712, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:39:05,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:05,200 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 712, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:39:16,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:39:16,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:39:16,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.04 seconds 2025-02-14 23:39:16,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:16,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17930.04 MB 2025-02-14 23:39:16,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20449.76 MB 2025-02-14 23:39:16,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2519.73 MB 2025-02-14 23:39:16,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52221.18 MB 2025-02-14 23:39:16,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22806.53 MB 2025-02-14 23:39:16,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29414.65 MB 2025-02-14 23:39:16,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29439.84 MB 2025-02-14 23:39:16,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:39:16,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:39:16,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 23:39:16,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:16,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20449.76 MB 2025-02-14 23:39:16,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19480.37 MB 2025-02-14 23:39:16,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -969.40 MB 2025-02-14 23:39:16,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22806.53 MB 2025-02-14 23:39:16,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34129.05 MB 2025-02-14 23:39:16,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11322.52 MB 2025-02-14 23:39:16,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29305.72 MB 2025-02-14 23:39:18,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:39:18,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:39:18,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:39:18,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19480.37 MB 2025-02-14 23:39:18,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20011.21 MB 2025-02-14 23:39:18,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:39:18,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34129.05 MB 2025-02-14 23:39:18,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24930.94 MB 2025-02-14 23:39:18,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9198.11 MB 2025-02-14 23:39:18,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23989.75 MB 2025-02-14 23:39:18,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:39:18,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:39:18,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:39:18,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20011.21 MB 2025-02-14 23:39:18,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21900.74 MB 2025-02-14 23:39:18,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:39:18,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24930.94 MB 2025-02-14 23:39:18,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25874.66 MB 2025-02-14 23:39:18,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 23:39:18,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23318.17 MB 2025-02-14 23:39:18,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:39:18,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:39:18,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:39:18,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21900.74 MB 2025-02-14 23:39:18,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24142.60 MB 2025-02-14 23:39:18,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:39:18,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25874.66 MB 2025-02-14 23:39:18,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32008.83 MB 2025-02-14 23:39:18,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:39:18,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29686.88 MB 2025-02-14 23:39:18,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:39:18,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:39:18,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:39:18,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20011.21 MB 2025-02-14 23:39:18,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24142.60 MB 2025-02-14 23:39:18,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:39:18,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24930.94 MB 2025-02-14 23:39:18,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32008.83 MB 2025-02-14 23:39:18,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 23:39:18,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29686.88 MB 2025-02-14 23:39:18,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:39:18,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:39:18,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:39:18,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25676.14 MB 2025-02-14 23:39:18,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26443.14 MB 2025-02-14 23:39:18,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:39:18,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32008.83 MB 2025-02-14 23:39:18,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32424.07 MB 2025-02-14 23:39:18,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:39:18,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27150.93 MB 2025-02-14 23:39:18,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:39:18,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:39:18,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:39:18,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26856.03 MB 2025-02-14 23:39:18,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27084.55 MB 2025-02-14 23:39:18,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.52 MB 2025-02-14 23:39:18,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32424.07 MB 2025-02-14 23:39:18,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32424.07 MB 2025-02-14 23:39:18,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:39:18,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27271.97 MB 2025-02-14 23:39:18,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:39:18,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:39:18,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.42 seconds 2025-02-14 23:39:18,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15449.37 MB 2025-02-14 23:39:18,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27285.62 MB 2025-02-14 23:39:18,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11836.25 MB 2025-02-14 23:39:18,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52221.18 MB 2025-02-14 23:39:18,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32424.07 MB 2025-02-14 23:39:18,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19797.11 MB 2025-02-14 23:39:18,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27285.62 MB 2025-02-14 23:39:18,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:39:18,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:39:18,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:39:18,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27285.62 MB 2025-02-14 23:39:18,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20453.76 MB 2025-02-14 23:39:18,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6831.86 MB 2025-02-14 23:39:18,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32424.07 MB 2025-02-14 23:39:18,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32424.07 MB 2025-02-14 23:39:18,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:39:18,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29797.29 MB 2025-02-14 23:39:18,912 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:39:18,912 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:39:18,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:39:18,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:39:18,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:39:18,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:18,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20453.76 MB 2025-02-14 23:39:18,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.78 MB 2025-02-14 23:39:18,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:39:18,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32424.07 MB 2025-02-14 23:39:18,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40814.77 MB 2025-02-14 23:39:18,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:39:18,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.78 MB 2025-02-14 23:39:19,075 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:39:19,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:19,077 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:39:19,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:19,078 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:39:19,082 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:39:19,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:19,083 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:39:19,084 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:39:31,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:31,879 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:39:31,884 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:39:31,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:31,888 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 637, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:39:31,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:31,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 637, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:39:41,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:39:41,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:39:41,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.86 seconds 2025-02-14 23:39:41,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:41,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17407.42 MB 2025-02-14 23:39:41,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19661.73 MB 2025-02-14 23:39:41,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2254.31 MB 2025-02-14 23:39:41,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53399.78 MB 2025-02-14 23:39:41,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24222.11 MB 2025-02-14 23:39:41,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29177.68 MB 2025-02-14 23:39:41,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28464.24 MB 2025-02-14 23:39:41,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:39:41,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:39:41,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 23:39:41,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:41,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19661.73 MB 2025-02-14 23:39:41,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19089.42 MB 2025-02-14 23:39:41,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -572.32 MB 2025-02-14 23:39:41,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24222.11 MB 2025-02-14 23:39:41,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31587.30 MB 2025-02-14 23:39:41,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7365.20 MB 2025-02-14 23:39:41,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28322.58 MB 2025-02-14 23:39:43,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:39:43,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:39:43,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:39:43,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:43,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19089.42 MB 2025-02-14 23:39:43,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19620.26 MB 2025-02-14 23:39:43,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:39:43,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31587.30 MB 2025-02-14 23:39:43,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25637.68 MB 2025-02-14 23:39:43,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5949.62 MB 2025-02-14 23:39:43,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23599.69 MB 2025-02-14 23:39:43,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:39:43,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:39:43,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:39:43,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:43,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19620.26 MB 2025-02-14 23:39:43,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21509.79 MB 2025-02-14 23:39:43,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:39:43,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25637.68 MB 2025-02-14 23:39:43,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25637.68 MB 2025-02-14 23:39:43,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:39:43,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22927.22 MB 2025-02-14 23:39:43,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:39:43,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:39:43,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:39:43,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:43,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21509.79 MB 2025-02-14 23:39:43,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23751.65 MB 2025-02-14 23:39:43,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:39:43,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25637.68 MB 2025-02-14 23:39:43,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31771.85 MB 2025-02-14 23:39:43,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:39:43,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29295.93 MB 2025-02-14 23:39:43,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:39:43,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:39:43,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:39:43,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:43,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19620.26 MB 2025-02-14 23:39:43,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23751.65 MB 2025-02-14 23:39:43,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:39:43,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25637.68 MB 2025-02-14 23:39:43,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31771.85 MB 2025-02-14 23:39:43,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:39:43,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29295.93 MB 2025-02-14 23:39:44,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:39:44,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:39:44,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:39:44,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:44,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25285.19 MB 2025-02-14 23:39:44,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26052.19 MB 2025-02-14 23:39:44,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:39:44,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31771.85 MB 2025-02-14 23:39:44,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32187.09 MB 2025-02-14 23:39:44,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:39:44,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26759.98 MB 2025-02-14 23:39:44,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:39:44,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:39:44,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:39:44,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:44,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26465.08 MB 2025-02-14 23:39:44,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26694.76 MB 2025-02-14 23:39:44,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.68 MB 2025-02-14 23:39:44,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32187.09 MB 2025-02-14 23:39:44,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32187.09 MB 2025-02-14 23:39:44,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:39:44,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26916.94 MB 2025-02-14 23:39:44,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:39:44,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:39:44,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.26 seconds 2025-02-14 23:39:44,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:44,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15188.07 MB 2025-02-14 23:39:44,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26895.83 MB 2025-02-14 23:39:44,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11707.76 MB 2025-02-14 23:39:44,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53399.78 MB 2025-02-14 23:39:44,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32187.09 MB 2025-02-14 23:39:44,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21212.69 MB 2025-02-14 23:39:44,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26916.94 MB 2025-02-14 23:39:44,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:39:44,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:39:44,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:39:44,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:44,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26895.83 MB 2025-02-14 23:39:44,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20192.25 MB 2025-02-14 23:39:44,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6703.58 MB 2025-02-14 23:39:44,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32187.09 MB 2025-02-14 23:39:44,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32187.09 MB 2025-02-14 23:39:44,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:39:44,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29407.50 MB 2025-02-14 23:39:44,433 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:39:44,434 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 23:39:44,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:39:44,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:39:44,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:39:44,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:39:44,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20192.25 MB 2025-02-14 23:39:44,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28631.27 MB 2025-02-14 23:39:44,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:39:44,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32187.09 MB 2025-02-14 23:39:44,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40577.79 MB 2025-02-14 23:39:44,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:39:44,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28631.27 MB 2025-02-14 23:39:44,595 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:39:44,597 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:44,597 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:39:44,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:44,598 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:39:44,602 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:39:44,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:39:44,603 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:39:44,604 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 23:40:40,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:40:40,109 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:40:40,114 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:40:40,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:40:40,118 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:40:40,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:40:40,119 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:40:43,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:40:43,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:40:43,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.04 seconds 2025-02-14 23:40:43,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:43,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14327.50 MB 2025-02-14 23:40:43,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15017.59 MB 2025-02-14 23:40:43,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.09 MB 2025-02-14 23:40:43,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53162.80 MB 2025-02-14 23:40:43,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 23:40:43,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33632.03 MB 2025-02-14 23:40:43,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24025.36 MB 2025-02-14 23:40:43,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:40:43,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:40:43,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:40:43,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:43,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15017.59 MB 2025-02-14 23:40:43,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15323.85 MB 2025-02-14 23:40:43,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.26 MB 2025-02-14 23:40:43,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 23:40:43,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19530.78 MB 2025-02-14 23:40:43,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:40:43,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17711.07 MB 2025-02-14 23:40:44,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:40:44,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:40:44,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-14 23:40:44,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15323.85 MB 2025-02-14 23:40:44,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15577.33 MB 2025-02-14 23:40:44,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 253.48 MB 2025-02-14 23:40:44,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19530.78 MB 2025-02-14 23:40:44,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19927.14 MB 2025-02-14 23:40:44,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 396.36 MB 2025-02-14 23:40:44,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19515.64 MB 2025-02-14 23:40:44,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:40:44,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:40:44,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:40:44,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15577.26 MB 2025-02-14 23:40:44,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16479.29 MB 2025-02-14 23:40:44,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.03 MB 2025-02-14 23:40:44,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19927.14 MB 2025-02-14 23:40:44,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19927.14 MB 2025-02-14 23:40:44,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:40:44,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17156.12 MB 2025-02-14 23:40:44,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:40:44,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:40:44,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 23:40:44,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16479.29 MB 2025-02-14 23:40:44,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17549.81 MB 2025-02-14 23:40:44,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1070.52 MB 2025-02-14 23:40:44,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19927.14 MB 2025-02-14 23:40:44,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21732.79 MB 2025-02-14 23:40:44,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1805.65 MB 2025-02-14 23:40:44,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20197.96 MB 2025-02-14 23:40:44,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:40:44,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:40:44,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 23:40:44,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15577.26 MB 2025-02-14 23:40:44,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17549.81 MB 2025-02-14 23:40:44,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1972.55 MB 2025-02-14 23:40:44,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19927.14 MB 2025-02-14 23:40:44,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21732.79 MB 2025-02-14 23:40:44,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1805.65 MB 2025-02-14 23:40:44,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20197.96 MB 2025-02-14 23:40:44,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:40:44,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:40:44,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 23:40:44,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18282.08 MB 2025-02-14 23:40:44,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18648.32 MB 2025-02-14 23:40:44,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 366.24 MB 2025-02-14 23:40:44,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21732.79 MB 2025-02-14 23:40:44,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21927.82 MB 2025-02-14 23:40:44,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-14 23:40:44,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18989.10 MB 2025-02-14 23:40:44,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:40:44,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:40:44,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:40:44,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18845.48 MB 2025-02-14 23:40:44,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19070.67 MB 2025-02-14 23:40:44,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.18 MB 2025-02-14 23:40:44,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21927.82 MB 2025-02-14 23:40:44,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21927.82 MB 2025-02-14 23:40:44,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:40:44,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.48 MB 2025-02-14 23:40:44,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:40:44,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:40:44,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.30 seconds 2025-02-14 23:40:44,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13648.10 MB 2025-02-14 23:40:44,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19271.40 MB 2025-02-14 23:40:44,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5623.29 MB 2025-02-14 23:40:44,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53162.80 MB 2025-02-14 23:40:44,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21927.82 MB 2025-02-14 23:40:44,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31234.98 MB 2025-02-14 23:40:44,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19271.40 MB 2025-02-14 23:40:44,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:40:44,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:40:44,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 23:40:44,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19271.40 MB 2025-02-14 23:40:44,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17660.83 MB 2025-02-14 23:40:44,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1610.57 MB 2025-02-14 23:40:44,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21927.82 MB 2025-02-14 23:40:44,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21927.82 MB 2025-02-14 23:40:44,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:40:44,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19271.41 MB 2025-02-14 23:40:44,733 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 23:40:44,733 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:40:44,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:40:44,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:40:44,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 23:40:44,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:40:44,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17660.83 MB 2025-02-14 23:40:44,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26085.78 MB 2025-02-14 23:40:44,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 23:40:44,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21927.82 MB 2025-02-14 23:40:44,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30303.85 MB 2025-02-14 23:40:44,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 23:40:44,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26085.78 MB 2025-02-14 23:40:44,988 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 23:40:44,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:40:44,991 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:40:44,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:40:44,993 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:40:45,000 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:40:45,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:40:45,002 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:40:45,002 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:41:24,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:41:24,428 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:41:24,433 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:41:24,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:41:24,437 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:41:24,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:41:24,438 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:41:43,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:41:43,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:41:43,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.44 seconds 2025-02-14 23:41:43,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:43,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-14 23:41:43,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-14 23:41:43,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-14 23:41:43,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38679.87 MB 2025-02-14 23:41:43,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35154.56 MB 2025-02-14 23:41:43,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3525.31 MB 2025-02-14 23:41:43,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.17 MB 2025-02-14 23:41:43,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:41:43,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:41:43,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 23:41:43,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:43,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-14 23:41:43,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-14 23:41:43,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-14 23:41:43,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35154.56 MB 2025-02-14 23:41:43,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44172.31 MB 2025-02-14 23:41:43,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9017.75 MB 2025-02-14 23:41:43,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39521.54 MB 2025-02-14 23:41:45,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:41:45,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:41:45,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:41:45,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:45,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-14 23:41:45,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-14 23:41:45,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:41:45,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44172.31 MB 2025-02-14 23:41:45,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-14 23:41:45,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13495.17 MB 2025-02-14 23:41:45,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26863.58 MB 2025-02-14 23:41:45,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:41:45,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:41:45,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:41:45,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:45,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 23:41:45,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-14 23:41:45,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:41:45,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 23:41:45,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30677.14 MB 2025-02-14 23:41:45,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:41:45,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-14 23:41:46,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:41:46,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:41:46,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:41:46,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:46,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-14 23:41:46,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 23:41:46,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:41:46,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 23:41:46,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34452.01 MB 2025-02-14 23:41:46,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 23:41:46,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 23:41:46,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:41:46,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:41:46,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:41:46,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:46,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 23:41:46,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 23:41:46,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:41:46,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30677.14 MB 2025-02-14 23:41:46,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34452.01 MB 2025-02-14 23:41:46,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 23:41:46,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 23:41:46,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:41:46,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:41:46,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:41:46,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:46,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-14 23:41:46,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-14 23:41:46,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:41:46,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34452.01 MB 2025-02-14 23:41:46,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 23:41:46,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:41:46,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-14 23:41:46,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:41:46,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:41:46,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:41:46,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:46,315 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-14 23:41:46,315 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.76 MB 2025-02-14 23:41:46,315 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.90 MB 2025-02-14 23:41:46,315 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34869.35 MB 2025-02-14 23:41:46,315 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 23:41:46,315 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:41:46,315 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30193.74 MB 2025-02-14 23:41:46,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:41:46,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:41:46,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.88 seconds 2025-02-14 23:41:46,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:46,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-14 23:41:46,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30159.83 MB 2025-02-14 23:41:46,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12783.76 MB 2025-02-14 23:41:46,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38679.87 MB 2025-02-14 23:41:46,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 23:41:46,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3810.53 MB 2025-02-14 23:41:46,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30193.74 MB 2025-02-14 23:41:46,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:41:46,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:41:46,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:41:46,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:46,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30159.83 MB 2025-02-14 23:41:46,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22380.46 MB 2025-02-14 23:41:46,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7779.37 MB 2025-02-14 23:41:46,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34869.35 MB 2025-02-14 23:41:46,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 23:41:46,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:41:46,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32671.50 MB 2025-02-14 23:41:46,603 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:41:46,603 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:41:46,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:41:46,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:41:46,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:41:46,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:41:46,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22380.46 MB 2025-02-14 23:41:46,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30819.48 MB 2025-02-14 23:41:46,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:41:46,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34869.35 MB 2025-02-14 23:41:46,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43260.05 MB 2025-02-14 23:41:46,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:41:46,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30819.48 MB 2025-02-14 23:41:46,768 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:41:46,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:41:46,770 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:41:46,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:41:46,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:41:46,775 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:41:46,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:41:46,776 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:41:46,776 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:42:48,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:42:48,967 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:42:48,972 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:42:48,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:42:48,976 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 875, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:42:48,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:42:48,977 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 875, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:43:02,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:43:02,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:43:02,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.39 seconds 2025-02-14 23:43:02,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:02,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19065.85 MB 2025-02-14 23:43:02,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22162.42 MB 2025-02-14 23:43:02,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3096.58 MB 2025-02-14 23:43:02,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55845.06 MB 2025-02-14 23:43:02,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26489.13 MB 2025-02-14 23:43:02,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29355.93 MB 2025-02-14 23:43:02,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31028.63 MB 2025-02-14 23:43:02,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:43:02,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:43:02,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 23:43:02,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:02,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22162.42 MB 2025-02-14 23:43:02,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20327.75 MB 2025-02-14 23:43:02,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1834.67 MB 2025-02-14 23:43:02,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26489.13 MB 2025-02-14 23:43:02,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36402.36 MB 2025-02-14 23:43:02,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9913.24 MB 2025-02-14 23:43:02,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31660.19 MB 2025-02-14 23:43:04,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:43:04,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:43:04,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 23:43:04,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:04,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20327.75 MB 2025-02-14 23:43:04,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20858.59 MB 2025-02-14 23:43:04,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:43:04,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36402.36 MB 2025-02-14 23:43:04,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29892.80 MB 2025-02-14 23:43:04,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6509.56 MB 2025-02-14 23:43:04,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24837.14 MB 2025-02-14 23:43:04,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:43:04,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:43:04,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:43:04,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:04,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20858.59 MB 2025-02-14 23:43:04,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22748.13 MB 2025-02-14 23:43:04,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:43:04,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29892.80 MB 2025-02-14 23:43:04,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29892.80 MB 2025-02-14 23:43:04,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:43:04,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24165.56 MB 2025-02-14 23:43:04,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:43:04,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:43:04,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:43:04,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:04,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22748.13 MB 2025-02-14 23:43:04,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24989.98 MB 2025-02-14 23:43:04,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:43:04,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29892.80 MB 2025-02-14 23:43:04,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33195.82 MB 2025-02-14 23:43:04,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 23:43:04,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30534.26 MB 2025-02-14 23:43:04,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:43:04,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:43:04,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:43:04,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:04,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20858.59 MB 2025-02-14 23:43:04,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24989.98 MB 2025-02-14 23:43:04,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:43:04,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29892.80 MB 2025-02-14 23:43:04,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33195.82 MB 2025-02-14 23:43:04,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 23:43:04,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30534.26 MB 2025-02-14 23:43:04,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:43:04,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:43:04,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:43:04,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:04,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26523.53 MB 2025-02-14 23:43:04,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27290.53 MB 2025-02-14 23:43:04,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:43:04,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33195.82 MB 2025-02-14 23:43:04,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 23:43:04,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:43:04,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27998.32 MB 2025-02-14 23:43:04,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:43:04,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:43:04,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:43:04,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:04,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27703.42 MB 2025-02-14 23:43:04,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27930.36 MB 2025-02-14 23:43:04,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.95 MB 2025-02-14 23:43:04,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 23:43:04,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 23:43:04,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:43:04,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28146.94 MB 2025-02-14 23:43:04,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:43:04,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:43:04,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.76 seconds 2025-02-14 23:43:04,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:04,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16017.28 MB 2025-02-14 23:43:04,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28131.44 MB 2025-02-14 23:43:04,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12114.16 MB 2025-02-14 23:43:04,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55845.06 MB 2025-02-14 23:43:04,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 23:43:04,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22231.91 MB 2025-02-14 23:43:04,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28146.94 MB 2025-02-14 23:43:05,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:43:05,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:43:05,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:43:05,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:05,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28131.44 MB 2025-02-14 23:43:05,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21021.67 MB 2025-02-14 23:43:05,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7109.77 MB 2025-02-14 23:43:05,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 23:43:05,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 23:43:05,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:43:05,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30643.10 MB 2025-02-14 23:43:05,021 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:43:05,021 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:43:05,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:43:05,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:43:05,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:43:05,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:43:05,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21021.67 MB 2025-02-14 23:43:05,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29460.69 MB 2025-02-14 23:43:05,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:43:05,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 23:43:05,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42003.86 MB 2025-02-14 23:43:05,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:43:05,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29460.69 MB 2025-02-14 23:43:05,184 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:43:05,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:43:05,185 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:43:05,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:43:05,186 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:43:05,191 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:43:05,192 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:43:05,192 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:43:05,192 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:43:59,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:43:59,471 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:43:59,476 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:43:59,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:43:59,480 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1254, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:43:59,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:43:59,481 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1254, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:44:18,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:44:18,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:44:18,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.32 seconds 2025-02-14 23:44:18,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:18,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21706.78 MB 2025-02-14 23:44:18,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26144.62 MB 2025-02-14 23:44:18,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4437.84 MB 2025-02-14 23:44:18,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54588.87 MB 2025-02-14 23:44:18,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35127.30 MB 2025-02-14 23:44:18,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19461.57 MB 2025-02-14 23:44:18,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35028.52 MB 2025-02-14 23:44:18,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:44:18,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:44:18,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 23:44:18,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:18,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26144.62 MB 2025-02-14 23:44:18,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22297.01 MB 2025-02-14 23:44:18,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3847.61 MB 2025-02-14 23:44:18,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35127.30 MB 2025-02-14 23:44:18,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43444.60 MB 2025-02-14 23:44:18,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8317.30 MB 2025-02-14 23:44:18,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38705.90 MB 2025-02-14 23:44:20,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:44:20,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:44:20,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 23:44:20,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:20,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22297.01 MB 2025-02-14 23:44:20,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22827.85 MB 2025-02-14 23:44:20,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:44:20,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43444.60 MB 2025-02-14 23:44:20,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30687.63 MB 2025-02-14 23:44:20,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12756.98 MB 2025-02-14 23:44:20,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26806.39 MB 2025-02-14 23:44:20,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:44:20,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:44:20,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:44:20,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:20,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22827.85 MB 2025-02-14 23:44:20,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24717.38 MB 2025-02-14 23:44:20,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:44:20,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30687.63 MB 2025-02-14 23:44:20,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30687.63 MB 2025-02-14 23:44:20,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:44:20,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26134.81 MB 2025-02-14 23:44:21,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:44:21,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:44:21,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:44:21,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:21,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24717.38 MB 2025-02-14 23:44:21,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26959.24 MB 2025-02-14 23:44:21,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:44:21,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30687.63 MB 2025-02-14 23:44:21,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34934.36 MB 2025-02-14 23:44:21,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 23:44:21,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32503.52 MB 2025-02-14 23:44:21,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:44:21,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:44:21,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:44:21,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:21,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22827.85 MB 2025-02-14 23:44:21,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26959.24 MB 2025-02-14 23:44:21,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:44:21,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30687.63 MB 2025-02-14 23:44:21,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34934.36 MB 2025-02-14 23:44:21,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 23:44:21,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32503.52 MB 2025-02-14 23:44:21,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:44:21,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:44:21,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:44:21,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:21,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28492.78 MB 2025-02-14 23:44:21,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29259.78 MB 2025-02-14 23:44:21,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:44:21,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34934.36 MB 2025-02-14 23:44:21,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35347.50 MB 2025-02-14 23:44:21,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 23:44:21,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29967.57 MB 2025-02-14 23:44:21,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:44:21,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:44:21,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:44:21,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:21,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29672.67 MB 2025-02-14 23:44:21,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29899.98 MB 2025-02-14 23:44:21,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.30 MB 2025-02-14 23:44:21,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35347.50 MB 2025-02-14 23:44:21,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35347.50 MB 2025-02-14 23:44:21,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:44:21,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30114.54 MB 2025-02-14 23:44:21,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:44:21,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:44:21,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.73 seconds 2025-02-14 23:44:21,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:21,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17337.74 MB 2025-02-14 23:44:21,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30100.83 MB 2025-02-14 23:44:21,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12763.08 MB 2025-02-14 23:44:21,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54588.87 MB 2025-02-14 23:44:21,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35347.50 MB 2025-02-14 23:44:21,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19241.37 MB 2025-02-14 23:44:21,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30114.54 MB 2025-02-14 23:44:21,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:44:21,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:44:21,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:44:21,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:21,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30100.83 MB 2025-02-14 23:44:21,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22338.70 MB 2025-02-14 23:44:21,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7762.12 MB 2025-02-14 23:44:21,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35347.50 MB 2025-02-14 23:44:21,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35347.50 MB 2025-02-14 23:44:21,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:44:21,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32609.73 MB 2025-02-14 23:44:21,497 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 23:44:21,497 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:44:21,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:44:21,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:44:21,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:44:21,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:44:21,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22338.70 MB 2025-02-14 23:44:21,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30768.86 MB 2025-02-14 23:44:21,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.16 MB 2025-02-14 23:44:21,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35347.50 MB 2025-02-14 23:44:21,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39537.61 MB 2025-02-14 23:44:21,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 23:44:21,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30768.86 MB 2025-02-14 23:44:21,659 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 23:44:21,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:44:21,661 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:44:21,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:44:21,662 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:44:21,666 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:44:21,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:44:21,667 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:44:21,668 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:44:55,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:44:56,000 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:44:56,005 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:44:56,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:44:56,009 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:44:56,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:44:56,010 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:45:14,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:45:14,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:45:14,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.82 seconds 2025-02-14 23:45:14,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:14,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-14 23:45:14,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-14 23:45:14,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 23:45:14,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47917.83 MB 2025-02-14 23:45:14,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30802.97 MB 2025-02-14 23:45:14,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17114.86 MB 2025-02-14 23:45:14,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-14 23:45:14,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:45:14,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:45:14,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 23:45:14,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:14,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-14 23:45:14,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22105.70 MB 2025-02-14 23:45:14,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3650.80 MB 2025-02-14 23:45:14,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30802.97 MB 2025-02-14 23:45:14,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45369.79 MB 2025-02-14 23:45:14,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14566.82 MB 2025-02-14 23:45:14,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38615.36 MB 2025-02-14 23:45:16,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:45:16,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:45:16,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 23:45:16,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:16,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22105.70 MB 2025-02-14 23:45:16,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22636.54 MB 2025-02-14 23:45:16,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:45:16,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45369.79 MB 2025-02-14 23:45:16,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24431.82 MB 2025-02-14 23:45:16,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20937.97 MB 2025-02-14 23:45:16,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26616.13 MB 2025-02-14 23:45:16,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:45:16,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:45:16,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:45:16,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:16,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22636.54 MB 2025-02-14 23:45:16,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24526.08 MB 2025-02-14 23:45:16,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:45:16,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24431.82 MB 2025-02-14 23:45:16,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27262.98 MB 2025-02-14 23:45:16,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 23:45:16,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25943.51 MB 2025-02-14 23:45:17,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:45:17,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:45:17,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:45:17,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:17,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24526.08 MB 2025-02-14 23:45:17,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26767.93 MB 2025-02-14 23:45:17,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:45:17,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27262.98 MB 2025-02-14 23:45:17,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33869.00 MB 2025-02-14 23:45:17,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:45:17,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32312.22 MB 2025-02-14 23:45:17,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:45:17,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:45:17,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:45:17,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:17,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22636.54 MB 2025-02-14 23:45:17,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26767.93 MB 2025-02-14 23:45:17,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:45:17,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24431.82 MB 2025-02-14 23:45:17,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33869.00 MB 2025-02-14 23:45:17,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 23:45:17,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32312.22 MB 2025-02-14 23:45:17,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:45:17,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:45:17,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:45:17,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:17,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28301.48 MB 2025-02-14 23:45:17,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29068.48 MB 2025-02-14 23:45:17,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:45:17,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33869.00 MB 2025-02-14 23:45:17,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-14 23:45:17,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 23:45:17,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29776.27 MB 2025-02-14 23:45:17,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:45:17,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:45:17,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:45:17,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:17,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29481.37 MB 2025-02-14 23:45:17,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29709.01 MB 2025-02-14 23:45:17,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.65 MB 2025-02-14 23:45:17,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34282.14 MB 2025-02-14 23:45:17,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-14 23:45:17,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:45:17,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29955.32 MB 2025-02-14 23:45:17,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:45:17,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:45:17,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.29 seconds 2025-02-14 23:45:17,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:17,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-14 23:45:17,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29909.03 MB 2025-02-14 23:45:17,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12700.20 MB 2025-02-14 23:45:17,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47917.83 MB 2025-02-14 23:45:17,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-14 23:45:17,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13635.68 MB 2025-02-14 23:45:17,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29955.32 MB 2025-02-14 23:45:17,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:45:17,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:45:17,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:45:17,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:17,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29909.03 MB 2025-02-14 23:45:17,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22197.31 MB 2025-02-14 23:45:17,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7711.72 MB 2025-02-14 23:45:17,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34282.14 MB 2025-02-14 23:45:17,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34282.14 MB 2025-02-14 23:45:17,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:45:17,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32407.95 MB 2025-02-14 23:45:17,593 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 23:45:17,593 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:45:17,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:45:17,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:45:17,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:45:17,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:45:17,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22197.31 MB 2025-02-14 23:45:17,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30592.52 MB 2025-02-14 23:45:17,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 23:45:17,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34282.14 MB 2025-02-14 23:45:17,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42628.81 MB 2025-02-14 23:45:17,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 23:45:17,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30592.52 MB 2025-02-14 23:45:17,760 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 23:45:17,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:45:17,761 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:45:17,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:45:17,762 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:45:17,767 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:45:17,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:45:17,768 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:45:17,768 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:46:24,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:46:24,468 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:46:24,473 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:46:24,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:46:24,477 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 734, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:46:24,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:46:24,478 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 734, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:46:35,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:46:35,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:46:35,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.21 seconds 2025-02-14 23:46:35,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:35,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18083.34 MB 2025-02-14 23:46:35,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20680.92 MB 2025-02-14 23:46:35,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2597.58 MB 2025-02-14 23:46:35,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50975.47 MB 2025-02-14 23:46:35,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23718.79 MB 2025-02-14 23:46:35,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27256.68 MB 2025-02-14 23:46:35,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29593.14 MB 2025-02-14 23:46:35,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:46:35,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:46:35,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 23:46:35,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:35,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20680.92 MB 2025-02-14 23:46:35,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19594.74 MB 2025-02-14 23:46:35,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1086.18 MB 2025-02-14 23:46:35,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23718.79 MB 2025-02-14 23:46:35,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33625.74 MB 2025-02-14 23:46:35,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9906.95 MB 2025-02-14 23:46:35,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29542.77 MB 2025-02-14 23:46:37,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:46:37,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:46:37,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 23:46:37,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:37,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19594.74 MB 2025-02-14 23:46:37,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20125.58 MB 2025-02-14 23:46:37,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:46:37,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33625.74 MB 2025-02-14 23:46:37,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25843.20 MB 2025-02-14 23:46:37,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7782.53 MB 2025-02-14 23:46:37,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24104.13 MB 2025-02-14 23:46:37,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:46:37,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:46:37,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:46:37,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:37,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20125.58 MB 2025-02-14 23:46:37,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22015.11 MB 2025-02-14 23:46:37,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:46:37,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25843.20 MB 2025-02-14 23:46:37,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25843.20 MB 2025-02-14 23:46:37,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:46:37,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23432.54 MB 2025-02-14 23:46:37,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:46:37,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:46:37,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:46:37,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:37,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22015.11 MB 2025-02-14 23:46:37,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.97 MB 2025-02-14 23:46:37,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:46:37,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25843.20 MB 2025-02-14 23:46:37,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31977.37 MB 2025-02-14 23:46:37,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:46:37,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29801.25 MB 2025-02-14 23:46:37,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:46:37,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:46:37,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:46:37,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:37,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20125.58 MB 2025-02-14 23:46:37,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.97 MB 2025-02-14 23:46:37,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:46:37,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25843.20 MB 2025-02-14 23:46:37,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31977.37 MB 2025-02-14 23:46:37,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:46:37,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29801.25 MB 2025-02-14 23:46:38,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:46:38,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:46:38,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:46:38,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:38,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25790.51 MB 2025-02-14 23:46:38,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26557.51 MB 2025-02-14 23:46:38,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:46:38,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31977.37 MB 2025-02-14 23:46:38,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32394.71 MB 2025-02-14 23:46:38,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:46:38,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27265.30 MB 2025-02-14 23:46:38,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:46:38,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:46:38,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:46:38,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:38,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26970.40 MB 2025-02-14 23:46:38,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27201.27 MB 2025-02-14 23:46:38,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.87 MB 2025-02-14 23:46:38,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32394.71 MB 2025-02-14 23:46:38,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32394.71 MB 2025-02-14 23:46:38,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:46:38,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27407.84 MB 2025-02-14 23:46:38,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:46:38,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:46:38,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.58 seconds 2025-02-14 23:46:38,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:38,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15526.02 MB 2025-02-14 23:46:38,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27402.34 MB 2025-02-14 23:46:38,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11876.32 MB 2025-02-14 23:46:38,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50975.47 MB 2025-02-14 23:46:38,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32394.71 MB 2025-02-14 23:46:38,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18580.77 MB 2025-02-14 23:46:38,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27407.84 MB 2025-02-14 23:46:38,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:46:38,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:46:38,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:46:38,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:38,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27402.34 MB 2025-02-14 23:46:38,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20530.41 MB 2025-02-14 23:46:38,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6871.93 MB 2025-02-14 23:46:38,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32394.71 MB 2025-02-14 23:46:38,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32394.71 MB 2025-02-14 23:46:38,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:46:38,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29914.01 MB 2025-02-14 23:46:38,344 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:46:38,344 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 23:46:38,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:46:38,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:46:38,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:46:38,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:46:38,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20530.41 MB 2025-02-14 23:46:38,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28969.43 MB 2025-02-14 23:46:38,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:46:38,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32394.71 MB 2025-02-14 23:46:38,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40785.41 MB 2025-02-14 23:46:38,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:46:38,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28969.43 MB 2025-02-14 23:46:38,506 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:46:38,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:46:38,508 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:46:38,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:46:38,509 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:46:38,513 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:46:38,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:46:38,515 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:46:38,515 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 23:48:05,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:48:05,142 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:48:05,147 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:48:05,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:48:05,151 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1533, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:48:05,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:48:05,152 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1533, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:48:28,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:48:28,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:48:28,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.50 seconds 2025-02-14 23:48:28,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:28,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23650.90 MB 2025-02-14 23:48:28,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29076.23 MB 2025-02-14 23:48:28,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5425.33 MB 2025-02-14 23:48:28,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53370.42 MB 2025-02-14 23:48:28,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36119.25 MB 2025-02-14 23:48:28,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17251.17 MB 2025-02-14 23:48:28,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37879.41 MB 2025-02-14 23:48:28,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:48:28,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:48:28,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 23:48:28,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:28,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29076.23 MB 2025-02-14 23:48:28,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23747.44 MB 2025-02-14 23:48:28,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5328.79 MB 2025-02-14 23:48:28,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36119.25 MB 2025-02-14 23:48:28,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49295.65 MB 2025-02-14 23:48:28,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13176.41 MB 2025-02-14 23:48:28,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44133.46 MB 2025-02-14 23:48:30,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:48:30,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:48:30,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:48:30,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:30,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23747.44 MB 2025-02-14 23:48:30,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24278.28 MB 2025-02-14 23:48:30,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:48:30,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49295.65 MB 2025-02-14 23:48:30,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30693.92 MB 2025-02-14 23:48:30,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18601.74 MB 2025-02-14 23:48:30,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28256.83 MB 2025-02-14 23:48:30,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:48:30,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:48:30,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:48:30,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:30,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.28 MB 2025-02-14 23:48:30,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26167.82 MB 2025-02-14 23:48:30,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:48:30,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30693.92 MB 2025-02-14 23:48:30,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30693.92 MB 2025-02-14 23:48:30,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:48:30,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27585.24 MB 2025-02-14 23:48:30,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:48:30,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:48:30,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:48:30,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:30,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26167.82 MB 2025-02-14 23:48:30,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28409.67 MB 2025-02-14 23:48:30,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:48:30,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30693.92 MB 2025-02-14 23:48:30,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35884.37 MB 2025-02-14 23:48:30,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 23:48:30,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33953.95 MB 2025-02-14 23:48:30,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:48:30,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:48:30,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:48:30,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:30,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.28 MB 2025-02-14 23:48:30,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28409.67 MB 2025-02-14 23:48:30,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:48:30,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30693.92 MB 2025-02-14 23:48:30,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35884.37 MB 2025-02-14 23:48:30,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 23:48:30,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33953.95 MB 2025-02-14 23:48:31,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:48:31,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:48:31,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:48:31,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:31,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29943.21 MB 2025-02-14 23:48:31,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30710.22 MB 2025-02-14 23:48:31,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:48:31,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35884.37 MB 2025-02-14 23:48:31,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36299.60 MB 2025-02-14 23:48:31,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:48:31,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31418.00 MB 2025-02-14 23:48:31,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:48:31,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:48:31,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:48:31,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:31,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31123.10 MB 2025-02-14 23:48:31,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31351.52 MB 2025-02-14 23:48:31,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.42 MB 2025-02-14 23:48:31,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36299.60 MB 2025-02-14 23:48:31,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36299.60 MB 2025-02-14 23:48:31,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:48:31,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31542.64 MB 2025-02-14 23:48:31,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:48:31,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:48:31,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.96 seconds 2025-02-14 23:48:31,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:31,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18309.80 MB 2025-02-14 23:48:31,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31551.98 MB 2025-02-14 23:48:31,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13242.18 MB 2025-02-14 23:48:31,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53370.42 MB 2025-02-14 23:48:31,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36299.60 MB 2025-02-14 23:48:31,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17070.82 MB 2025-02-14 23:48:31,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31551.98 MB 2025-02-14 23:48:31,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:48:31,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:48:31,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:48:31,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:31,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31551.98 MB 2025-02-14 23:48:31,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23304.69 MB 2025-02-14 23:48:31,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8247.29 MB 2025-02-14 23:48:31,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36299.60 MB 2025-02-14 23:48:31,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36299.60 MB 2025-02-14 23:48:31,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:48:31,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34055.99 MB 2025-02-14 23:48:31,399 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 23:48:31,400 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:48:31,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:48:31,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:48:31,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:48:31,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:48:31,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23304.69 MB 2025-02-14 23:48:31,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31718.15 MB 2025-02-14 23:48:31,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.46 MB 2025-02-14 23:48:31,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36299.60 MB 2025-02-14 23:48:31,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40481.33 MB 2025-02-14 23:48:31,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 23:48:31,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31718.15 MB 2025-02-14 23:48:31,561 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 23:48:31,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:48:31,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:48:31,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:48:31,564 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:48:31,568 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:48:31,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:48:31,569 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:48:31,570 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:49:41,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:49:41,604 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:49:41,612 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:49:41,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:49:41,620 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1901, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:49:41,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:49:41,621 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1901, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:50:11,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:50:11,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:50:11,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.42 seconds 2025-02-14 23:50:11,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:11,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26215.18 MB 2025-02-14 23:50:11,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32942.84 MB 2025-02-14 23:50:11,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6727.66 MB 2025-02-14 23:50:11,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48844.77 MB 2025-02-14 23:50:11,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37398.51 MB 2025-02-14 23:50:11,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11446.26 MB 2025-02-14 23:50:11,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41802.65 MB 2025-02-14 23:50:11,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:50:11,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:50:11,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 23:50:11,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:11,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32942.84 MB 2025-02-14 23:50:11,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25660.56 MB 2025-02-14 23:50:11,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7282.29 MB 2025-02-14 23:50:11,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37398.51 MB 2025-02-14 23:50:11,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61108.91 MB 2025-02-14 23:50:11,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23710.40 MB 2025-02-14 23:50:11,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51738.17 MB 2025-02-14 23:50:13,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:50:13,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:50:13,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 23:50:13,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25660.56 MB 2025-02-14 23:50:13,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26191.40 MB 2025-02-14 23:50:13,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:50:13,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61108.91 MB 2025-02-14 23:50:13,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32086.43 MB 2025-02-14 23:50:13,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29022.49 MB 2025-02-14 23:50:13,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30169.95 MB 2025-02-14 23:50:13,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:50:13,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:50:13,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:50:13,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26191.40 MB 2025-02-14 23:50:13,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28080.93 MB 2025-02-14 23:50:13,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:50:13,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32086.43 MB 2025-02-14 23:50:13,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32086.43 MB 2025-02-14 23:50:13,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:50:13,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29498.36 MB 2025-02-14 23:50:13,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:50:13,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:50:13,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:50:13,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28080.93 MB 2025-02-14 23:50:13,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.79 MB 2025-02-14 23:50:13,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:50:13,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32086.43 MB 2025-02-14 23:50:13,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37748.74 MB 2025-02-14 23:50:13,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:50:13,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35867.07 MB 2025-02-14 23:50:13,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:50:13,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:50:13,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:50:13,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26191.40 MB 2025-02-14 23:50:13,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.79 MB 2025-02-14 23:50:13,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:50:13,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32086.43 MB 2025-02-14 23:50:13,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37748.74 MB 2025-02-14 23:50:13,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 23:50:13,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35867.07 MB 2025-02-14 23:50:13,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:50:13,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:50:13,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:50:13,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31856.33 MB 2025-02-14 23:50:13,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32623.33 MB 2025-02-14 23:50:13,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:50:13,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37748.74 MB 2025-02-14 23:50:13,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-14 23:50:13,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 23:50:13,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33331.12 MB 2025-02-14 23:50:13,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:50:13,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:50:13,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:50:13,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33036.22 MB 2025-02-14 23:50:13,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33264.32 MB 2025-02-14 23:50:13,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 23:50:13,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-14 23:50:13,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-14 23:50:13,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:50:13,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33495.02 MB 2025-02-14 23:50:13,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:50:13,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:50:13,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.89 seconds 2025-02-14 23:50:13,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19591.94 MB 2025-02-14 23:50:13,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33464.80 MB 2025-02-14 23:50:13,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13872.86 MB 2025-02-14 23:50:13,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48844.77 MB 2025-02-14 23:50:13,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-14 23:50:13,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10682.89 MB 2025-02-14 23:50:13,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33495.02 MB 2025-02-14 23:50:13,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:50:13,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:50:13,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:50:13,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33464.80 MB 2025-02-14 23:50:13,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24580.42 MB 2025-02-14 23:50:13,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8884.39 MB 2025-02-14 23:50:13,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-14 23:50:13,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-14 23:50:13,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:50:13,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35963.26 MB 2025-02-14 23:50:13,806 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 23:50:13,806 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:50:13,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:50:13,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:50:13,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:50:13,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:13,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24580.42 MB 2025-02-14 23:50:13,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32975.63 MB 2025-02-14 23:50:13,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 23:50:13,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-14 23:50:13,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46508.54 MB 2025-02-14 23:50:13,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 23:50:13,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32975.63 MB 2025-02-14 23:50:13,970 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 23:50:13,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:13,971 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:50:13,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:13,972 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:50:13,977 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:50:13,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:13,978 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:50:13,978 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:50:35,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:35,944 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:50:35,949 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:50:35,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:35,953 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1351, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:50:35,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:35,954 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1351, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:50:56,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:50:56,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:50:56,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.95 seconds 2025-02-14 23:50:56,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:56,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22382.69 MB 2025-02-14 23:50:56,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27164.20 MB 2025-02-14 23:50:56,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4781.51 MB 2025-02-14 23:50:56,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54855.20 MB 2025-02-14 23:50:56,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35418.80 MB 2025-02-14 23:50:56,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19436.40 MB 2025-02-14 23:50:56,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36157.42 MB 2025-02-14 23:50:56,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:50:56,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:50:56,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:50:56,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:56,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27164.20 MB 2025-02-14 23:50:56,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22801.28 MB 2025-02-14 23:50:56,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4362.92 MB 2025-02-14 23:50:56,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35418.80 MB 2025-02-14 23:50:56,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45705.33 MB 2025-02-14 23:50:56,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10286.53 MB 2025-02-14 23:50:56,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40964.27 MB 2025-02-14 23:50:58,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:50:58,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:50:58,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 23:50:58,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:58,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22801.28 MB 2025-02-14 23:50:58,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23332.12 MB 2025-02-14 23:50:58,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:50:58,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45705.33 MB 2025-02-14 23:50:58,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26463.96 MB 2025-02-14 23:50:58,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19241.37 MB 2025-02-14 23:50:58,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27310.67 MB 2025-02-14 23:50:58,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:50:58,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:50:58,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:50:58,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:58,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23332.12 MB 2025-02-14 23:50:58,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25221.65 MB 2025-02-14 23:50:58,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:50:58,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26463.96 MB 2025-02-14 23:50:58,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28351.40 MB 2025-02-14 23:50:58,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 23:50:58,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26639.08 MB 2025-02-14 23:50:59,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:50:59,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:50:59,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:50:59,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:59,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25221.65 MB 2025-02-14 23:50:59,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27463.51 MB 2025-02-14 23:50:59,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:50:59,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28351.40 MB 2025-02-14 23:50:59,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34957.43 MB 2025-02-14 23:50:59,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 23:50:59,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33007.79 MB 2025-02-14 23:50:59,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:50:59,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:50:59,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:50:59,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:59,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23332.12 MB 2025-02-14 23:50:59,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27463.51 MB 2025-02-14 23:50:59,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:50:59,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26463.96 MB 2025-02-14 23:50:59,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34957.43 MB 2025-02-14 23:50:59,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 23:50:59,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33007.79 MB 2025-02-14 23:50:59,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:50:59,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:50:59,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 23:50:59,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:59,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28997.05 MB 2025-02-14 23:50:59,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29764.05 MB 2025-02-14 23:50:59,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:50:59,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34957.43 MB 2025-02-14 23:50:59,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35370.57 MB 2025-02-14 23:50:59,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 23:50:59,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30471.84 MB 2025-02-14 23:50:59,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:50:59,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:50:59,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:50:59,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:59,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30176.94 MB 2025-02-14 23:50:59,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30404.94 MB 2025-02-14 23:50:59,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.00 MB 2025-02-14 23:50:59,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35370.57 MB 2025-02-14 23:50:59,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35370.57 MB 2025-02-14 23:50:59,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:50:59,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30629.44 MB 2025-02-14 23:50:59,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:50:59,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:50:59,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.41 seconds 2025-02-14 23:50:59,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:59,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17675.70 MB 2025-02-14 23:50:59,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30604.86 MB 2025-02-14 23:50:59,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12929.16 MB 2025-02-14 23:50:59,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54855.20 MB 2025-02-14 23:50:59,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35370.57 MB 2025-02-14 23:50:59,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19484.64 MB 2025-02-14 23:50:59,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30629.44 MB 2025-02-14 23:50:59,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:50:59,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:50:59,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:50:59,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:59,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30604.86 MB 2025-02-14 23:50:59,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22662.75 MB 2025-02-14 23:50:59,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7942.11 MB 2025-02-14 23:50:59,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35370.57 MB 2025-02-14 23:50:59,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35370.57 MB 2025-02-14 23:50:59,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:50:59,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33102.09 MB 2025-02-14 23:50:59,651 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-14 23:50:59,651 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:50:59,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:50:59,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:50:59,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:50:59,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:50:59,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22662.75 MB 2025-02-14 23:50:59,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31053.79 MB 2025-02-14 23:50:59,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.04 MB 2025-02-14 23:50:59,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35370.57 MB 2025-02-14 23:50:59,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43713.04 MB 2025-02-14 23:50:59,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 23:50:59,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31053.79 MB 2025-02-14 23:50:59,812 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-14 23:50:59,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:59,813 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:50:59,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:59,814 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:50:59,819 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:50:59,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:50:59,820 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:50:59,820 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:51:57,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:51:57,320 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:51:57,325 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:51:57,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:51:57,329 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 406, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:51:57,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:51:57,330 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 406, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:52:03,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:52:03,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:52:03,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.32 seconds 2025-02-14 23:52:03,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:03,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15797.78 MB 2025-02-14 23:52:03,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17234.59 MB 2025-02-14 23:52:03,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1436.81 MB 2025-02-14 23:52:03,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52055.51 MB 2025-02-14 23:52:03,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25163.73 MB 2025-02-14 23:52:03,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26891.78 MB 2025-02-14 23:52:03,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26175.12 MB 2025-02-14 23:52:03,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:52:03,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:52:03,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 23:52:03,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:03,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17234.59 MB 2025-02-14 23:52:03,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17790.20 MB 2025-02-14 23:52:03,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 555.60 MB 2025-02-14 23:52:03,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25163.73 MB 2025-02-14 23:52:03,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26531.07 MB 2025-02-14 23:52:03,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1367.34 MB 2025-02-14 23:52:03,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22677.60 MB 2025-02-14 23:52:05,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:52:05,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:52:05,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.84 seconds 2025-02-14 23:52:05,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:05,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17790.20 MB 2025-02-14 23:52:05,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18302.46 MB 2025-02-14 23:52:05,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 512.26 MB 2025-02-14 23:52:05,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26531.07 MB 2025-02-14 23:52:05,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25587.35 MB 2025-02-14 23:52:05,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 23:52:05,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22299.58 MB 2025-02-14 23:52:05,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:52:05,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:52:05,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:52:05,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:05,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18302.46 MB 2025-02-14 23:52:05,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20125.93 MB 2025-02-14 23:52:05,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1823.47 MB 2025-02-14 23:52:05,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25587.35 MB 2025-02-14 23:52:05,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25587.35 MB 2025-02-14 23:52:05,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:52:05,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21493.75 MB 2025-02-14 23:52:05,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:52:05,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:52:05,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 23:52:05,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:05,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20125.93 MB 2025-02-14 23:52:05,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22289.33 MB 2025-02-14 23:52:05,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2163.39 MB 2025-02-14 23:52:05,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25587.35 MB 2025-02-14 23:52:05,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30148.66 MB 2025-02-14 23:52:05,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4561.31 MB 2025-02-14 23:52:05,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27639.56 MB 2025-02-14 23:52:05,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:52:05,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:52:05,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:52:05,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:05,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18302.46 MB 2025-02-14 23:52:05,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22289.33 MB 2025-02-14 23:52:05,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3986.87 MB 2025-02-14 23:52:05,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25587.35 MB 2025-02-14 23:52:05,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30148.66 MB 2025-02-14 23:52:05,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4561.31 MB 2025-02-14 23:52:05,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27639.56 MB 2025-02-14 23:52:05,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:52:05,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:52:05,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 23:52:05,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:05,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23769.19 MB 2025-02-14 23:52:05,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24509.35 MB 2025-02-14 23:52:05,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 740.16 MB 2025-02-14 23:52:05,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30148.66 MB 2025-02-14 23:52:05,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30551.31 MB 2025-02-14 23:52:05,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 23:52:05,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25192.37 MB 2025-02-14 23:52:05,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:52:05,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:52:05,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:52:05,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:05,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24907.79 MB 2025-02-14 23:52:05,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25114.31 MB 2025-02-14 23:52:05,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.52 MB 2025-02-14 23:52:05,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30551.31 MB 2025-02-14 23:52:05,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30555.50 MB 2025-02-14 23:52:05,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 23:52:05,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25303.22 MB 2025-02-14 23:52:05,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:52:05,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:52:05,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.59 seconds 2025-02-14 23:52:05,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:05,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14383.24 MB 2025-02-14 23:52:05,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25315.38 MB 2025-02-14 23:52:05,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10932.14 MB 2025-02-14 23:52:05,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52055.51 MB 2025-02-14 23:52:05,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30555.50 MB 2025-02-14 23:52:05,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21500.00 MB 2025-02-14 23:52:05,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25315.38 MB 2025-02-14 23:52:06,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:52:06,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:52:06,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:52:06,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:06,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25315.38 MB 2025-02-14 23:52:06,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19321.04 MB 2025-02-14 23:52:06,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5994.34 MB 2025-02-14 23:52:06,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30555.50 MB 2025-02-14 23:52:06,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30555.50 MB 2025-02-14 23:52:06,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:52:06,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28027.98 MB 2025-02-14 23:52:06,208 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 23:52:06,208 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:52:06,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:52:06,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:52:06,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:52:06,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:52:06,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19321.04 MB 2025-02-14 23:52:06,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27760.06 MB 2025-02-14 23:52:06,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 23:52:06,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30555.50 MB 2025-02-14 23:52:06,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38946.21 MB 2025-02-14 23:52:06,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 23:52:06,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27760.06 MB 2025-02-14 23:52:06,371 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 23:52:06,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:52:06,372 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:52:06,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:52:06,373 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:52:06,378 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:52:06,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:52:06,379 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:52:06,379 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:53:00,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:53:00,012 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:53:00,017 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:53:00,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:53:00,021 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1190, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:53:00,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:53:00,022 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1190, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:53:18,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:53:18,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:53:18,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.30 seconds 2025-02-14 23:53:18,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:18,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21260.82 MB 2025-02-14 23:53:18,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25472.16 MB 2025-02-14 23:53:18,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4211.34 MB 2025-02-14 23:53:18,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51531.22 MB 2025-02-14 23:53:18,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29798.43 MB 2025-02-14 23:53:18,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21732.79 MB 2025-02-14 23:53:18,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34356.07 MB 2025-02-14 23:53:18,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:53:18,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:53:18,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:53:18,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:18,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25472.16 MB 2025-02-14 23:53:18,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21964.29 MB 2025-02-14 23:53:18,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3507.87 MB 2025-02-14 23:53:18,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29798.43 MB 2025-02-14 23:53:18,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45300.58 MB 2025-02-14 23:53:18,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15502.15 MB 2025-02-14 23:53:18,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38048.85 MB 2025-02-14 23:53:20,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:53:20,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:53:20,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 23:53:20,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:20,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21964.29 MB 2025-02-14 23:53:20,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22495.13 MB 2025-02-14 23:53:20,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:53:20,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45300.58 MB 2025-02-14 23:53:20,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27000.83 MB 2025-02-14 23:53:20,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18299.75 MB 2025-02-14 23:53:20,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26473.68 MB 2025-02-14 23:53:20,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:53:20,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:53:20,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:53:20,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:20,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22495.13 MB 2025-02-14 23:53:20,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24384.67 MB 2025-02-14 23:53:20,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:53:20,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27000.83 MB 2025-02-14 23:53:20,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27944.55 MB 2025-02-14 23:53:20,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 23:53:20,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25802.09 MB 2025-02-14 23:53:20,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:53:20,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:53:20,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:53:20,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:20,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24384.67 MB 2025-02-14 23:53:20,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26626.52 MB 2025-02-14 23:53:20,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:53:20,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27944.55 MB 2025-02-14 23:53:20,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34078.72 MB 2025-02-14 23:53:20,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:53:20,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32170.80 MB 2025-02-14 23:53:20,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:53:20,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:53:20,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:53:20,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:20,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22495.13 MB 2025-02-14 23:53:20,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26626.52 MB 2025-02-14 23:53:20,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:53:20,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27000.83 MB 2025-02-14 23:53:20,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34078.72 MB 2025-02-14 23:53:20,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 23:53:20,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32170.80 MB 2025-02-14 23:53:20,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:53:20,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:53:20,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:53:20,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:20,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28160.06 MB 2025-02-14 23:53:20,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28927.07 MB 2025-02-14 23:53:20,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:53:20,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34078.72 MB 2025-02-14 23:53:20,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:53:20,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 23:53:20,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29634.85 MB 2025-02-14 23:53:20,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:53:20,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:53:20,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:53:20,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:20,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29339.95 MB 2025-02-14 23:53:20,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29568.49 MB 2025-02-14 23:53:20,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.53 MB 2025-02-14 23:53:20,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 23:53:20,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:53:20,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:53:20,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29810.07 MB 2025-02-14 23:53:20,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:53:20,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:53:20,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.71 seconds 2025-02-14 23:53:20,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:20,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17114.76 MB 2025-02-14 23:53:20,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29769.09 MB 2025-02-14 23:53:20,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12654.33 MB 2025-02-14 23:53:20,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51531.22 MB 2025-02-14 23:53:20,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:53:20,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17037.26 MB 2025-02-14 23:53:20,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29810.07 MB 2025-02-14 23:53:21,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:53:21,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:53:21,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:53:21,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:21,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29769.09 MB 2025-02-14 23:53:21,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22111.91 MB 2025-02-14 23:53:21,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7657.18 MB 2025-02-14 23:53:21,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 23:53:21,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:53:21,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:53:21,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32274.92 MB 2025-02-14 23:53:21,024 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 23:53:21,025 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:53:21,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:53:21,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:53:21,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:53:21,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:53:21,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22111.91 MB 2025-02-14 23:53:21,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.99 MB 2025-02-14 23:53:21,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 23:53:21,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 23:53:21,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42865.79 MB 2025-02-14 23:53:21,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 23:53:21,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30530.99 MB 2025-02-14 23:53:21,186 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 23:53:21,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:53:21,188 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:53:21,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:53:21,189 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:53:21,193 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:53:21,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:53:21,194 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:53:21,194 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:54:32,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:54:32,158 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:54:32,163 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:54:32,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:54:32,167 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:54:32,168 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:54:32,168 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:54:50,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:54:50,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:54:50,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.19 seconds 2025-02-14 23:54:50,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:50,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21232.94 MB 2025-02-14 23:54:50,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25430.13 MB 2025-02-14 23:54:50,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4197.19 MB 2025-02-14 23:54:50,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51237.62 MB 2025-02-14 23:54:50,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29785.85 MB 2025-02-14 23:54:50,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21451.77 MB 2025-02-14 23:54:50,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34328.65 MB 2025-02-14 23:54:50,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:54:50,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:54:50,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.32 seconds 2025-02-14 23:54:50,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:50,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25430.13 MB 2025-02-14 23:54:50,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21943.49 MB 2025-02-14 23:54:50,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3486.64 MB 2025-02-14 23:54:50,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29785.85 MB 2025-02-14 23:54:50,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45323.65 MB 2025-02-14 23:54:50,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15537.80 MB 2025-02-14 23:54:50,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37985.59 MB 2025-02-14 23:54:52,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:54:52,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:54:52,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 23:54:52,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:52,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21943.49 MB 2025-02-14 23:54:52,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22474.34 MB 2025-02-14 23:54:52,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:54:52,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45323.65 MB 2025-02-14 23:54:52,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27002.93 MB 2025-02-14 23:54:52,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18320.72 MB 2025-02-14 23:54:52,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26452.88 MB 2025-02-14 23:54:52,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:54:52,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:54:52,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:54:52,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:52,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22474.34 MB 2025-02-14 23:54:52,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24363.87 MB 2025-02-14 23:54:52,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:54:52,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27002.93 MB 2025-02-14 23:54:52,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27946.65 MB 2025-02-14 23:54:52,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 23:54:52,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25781.30 MB 2025-02-14 23:54:52,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:54:52,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:54:52,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:54:52,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:52,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24363.87 MB 2025-02-14 23:54:52,821 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26605.73 MB 2025-02-14 23:54:52,821 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:54:52,821 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27946.65 MB 2025-02-14 23:54:52,821 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34080.82 MB 2025-02-14 23:54:52,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:54:52,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32150.01 MB 2025-02-14 23:54:52,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:54:52,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:54:52,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:54:52,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:52,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22474.34 MB 2025-02-14 23:54:52,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26605.73 MB 2025-02-14 23:54:52,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:54:52,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27002.93 MB 2025-02-14 23:54:52,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34080.82 MB 2025-02-14 23:54:52,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 23:54:52,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32150.01 MB 2025-02-14 23:54:52,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:54:52,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:54:52,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:54:52,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:52,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28139.27 MB 2025-02-14 23:54:52,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28906.27 MB 2025-02-14 23:54:52,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:54:52,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34080.82 MB 2025-02-14 23:54:52,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:54:52,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 23:54:52,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29614.06 MB 2025-02-14 23:54:53,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:54:53,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:54:53,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:54:53,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:53,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29319.16 MB 2025-02-14 23:54:53,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29545.61 MB 2025-02-14 23:54:53,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.45 MB 2025-02-14 23:54:53,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 23:54:53,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:54:53,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:54:53,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.47 MB 2025-02-14 23:54:53,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:54:53,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:54:53,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.85 seconds 2025-02-14 23:54:53,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:53,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17100.83 MB 2025-02-14 23:54:53,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29745.63 MB 2025-02-14 23:54:53,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12644.80 MB 2025-02-14 23:54:53,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51237.62 MB 2025-02-14 23:54:53,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:54:53,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16743.66 MB 2025-02-14 23:54:53,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.47 MB 2025-02-14 23:54:53,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:54:53,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:54:53,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:54:53,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:53,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29745.63 MB 2025-02-14 23:54:53,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32744.28 MB 2025-02-14 23:54:53,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2998.65 MB 2025-02-14 23:54:53,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 23:54:53,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34493.96 MB 2025-02-14 23:54:53,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:54:53,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33044.06 MB 2025-02-14 23:54:53,303 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 23:54:53,303 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 23:54:53,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:54:53,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:54:53,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:54:53,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:54:53,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32744.28 MB 2025-02-14 23:54:53,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41139.49 MB 2025-02-14 23:54:53,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 23:54:53,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34493.96 MB 2025-02-14 23:54:53,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44927.29 MB 2025-02-14 23:54:53,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10433.33 MB 2025-02-14 23:54:53,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41139.49 MB 2025-02-14 23:54:53,465 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 23:54:53,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:54:53,467 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:54:53,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:54:53,468 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:54:53,472 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:54:53,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:54:53,473 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:54:53,473 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 23:56:07,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:56:07,534 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:56:07,541 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:56:07,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:56:07,547 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1327, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:56:07,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:56:07,549 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1327, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:56:27,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:56:27,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:56:27,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.35 seconds 2025-02-14 23:56:27,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:27,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42563.95 MB 2025-02-14 23:56:27,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47260.13 MB 2025-02-14 23:56:27,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4696.18 MB 2025-02-14 23:56:27,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55551.46 MB 2025-02-14 23:56:27,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55329.16 MB 2025-02-14 23:56:27,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -222.30 MB 2025-02-14 23:56:27,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56112.64 MB 2025-02-14 23:56:27,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:56:27,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:56:27,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 23:56:27,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:27,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47260.13 MB 2025-02-14 23:56:27,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43025.00 MB 2025-02-14 23:56:27,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4235.12 MB 2025-02-14 23:56:27,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55329.16 MB 2025-02-14 23:56:27,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65120.76 MB 2025-02-14 23:56:27,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9791.60 MB 2025-02-14 23:56:27,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60527.82 MB 2025-02-14 23:56:29,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:56:29,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:56:29,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 23:56:29,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:29,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43025.00 MB 2025-02-14 23:56:29,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43555.84 MB 2025-02-14 23:56:29,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:56:29,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65120.76 MB 2025-02-14 23:56:29,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46458.21 MB 2025-02-14 23:56:29,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18662.56 MB 2025-02-14 23:56:29,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47535.43 MB 2025-02-14 23:56:29,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:56:29,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:56:29,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:56:29,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:29,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43555.84 MB 2025-02-14 23:56:29,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45445.38 MB 2025-02-14 23:56:29,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:56:29,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46458.21 MB 2025-02-14 23:56:29,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49763.32 MB 2025-02-14 23:56:29,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-14 23:56:29,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46862.81 MB 2025-02-14 23:56:30,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:56:30,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:56:30,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 23:56:30,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:30,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45445.38 MB 2025-02-14 23:56:30,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47687.23 MB 2025-02-14 23:56:30,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:56:30,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49763.32 MB 2025-02-14 23:56:30,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55897.49 MB 2025-02-14 23:56:30,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 23:56:30,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53231.52 MB 2025-02-14 23:56:30,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:56:30,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:56:30,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:56:30,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:30,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43555.84 MB 2025-02-14 23:56:30,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47687.23 MB 2025-02-14 23:56:30,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:56:30,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46458.21 MB 2025-02-14 23:56:30,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55897.49 MB 2025-02-14 23:56:30,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-14 23:56:30,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53231.52 MB 2025-02-14 23:56:30,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:56:30,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:56:30,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 23:56:30,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:30,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49220.78 MB 2025-02-14 23:56:30,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49987.78 MB 2025-02-14 23:56:30,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:56:30,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55897.49 MB 2025-02-14 23:56:30,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56314.82 MB 2025-02-14 23:56:30,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:56:30,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50695.57 MB 2025-02-14 23:56:30,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:56:30,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:56:30,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:56:30,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:30,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50400.67 MB 2025-02-14 23:56:30,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50629.75 MB 2025-02-14 23:56:30,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-14 23:56:30,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56314.82 MB 2025-02-14 23:56:30,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56314.82 MB 2025-02-14 23:56:30,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:56:30,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50829.45 MB 2025-02-14 23:56:30,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:56:30,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:56:30,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.76 seconds 2025-02-14 23:56:30,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:30,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37940.45 MB 2025-02-14 23:56:30,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50829.99 MB 2025-02-14 23:56:30,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12889.54 MB 2025-02-14 23:56:30,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53276.05 MB 2025-02-14 23:56:30,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56314.82 MB 2025-02-14 23:56:30,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3038.77 MB 2025-02-14 23:56:30,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50829.99 MB 2025-02-14 23:56:30,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:56:30,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:56:30,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 23:56:30,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:30,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50829.99 MB 2025-02-14 23:56:30,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42932.24 MB 2025-02-14 23:56:30,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7897.75 MB 2025-02-14 23:56:30,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56314.82 MB 2025-02-14 23:56:30,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56314.82 MB 2025-02-14 23:56:30,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:56:30,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50829.99 MB 2025-02-14 23:56:30,591 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 23:56:30,591 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:56:30,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:56:30,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:56:30,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:56:30,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:56:30,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42932.24 MB 2025-02-14 23:56:30,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51338.06 MB 2025-02-14 23:56:30,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.82 MB 2025-02-14 23:56:30,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56314.82 MB 2025-02-14 23:56:30,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64669.88 MB 2025-02-14 23:56:30,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 23:56:30,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51338.06 MB 2025-02-14 23:56:30,756 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 23:56:30,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:56:30,757 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:56:30,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:56:30,758 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:56:30,763 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:56:30,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:56:30,764 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:56:30,764 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:57:19,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:57:19,603 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:57:19,611 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:57:19,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:57:19,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1692, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:57:19,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:57:19,620 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1692, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 23:57:45,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 23:57:45,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 23:57:45,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.19 seconds 2025-02-14 23:57:45,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:45,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45107.20 MB 2025-02-14 23:57:45,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51095.09 MB 2025-02-14 23:57:45,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5987.89 MB 2025-02-14 23:57:45,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73024.93 MB 2025-02-14 23:57:45,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58527.32 MB 2025-02-14 23:57:45,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14497.61 MB 2025-02-14 23:57:45,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60015.65 MB 2025-02-14 23:57:45,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 23:57:45,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 23:57:45,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 23:57:45,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:45,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51095.09 MB 2025-02-14 23:57:45,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44922.40 MB 2025-02-14 23:57:45,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6172.70 MB 2025-02-14 23:57:45,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58527.32 MB 2025-02-14 23:57:45,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77628.18 MB 2025-02-14 23:57:45,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19100.86 MB 2025-02-14 23:57:45,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68609.58 MB 2025-02-14 23:57:47,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 23:57:47,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 23:57:47,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 23:57:47,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:47,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44922.40 MB 2025-02-14 23:57:47,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45453.24 MB 2025-02-14 23:57:47,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 23:57:47,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77628.18 MB 2025-02-14 23:57:47,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53953.43 MB 2025-02-14 23:57:47,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23674.75 MB 2025-02-14 23:57:47,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49431.79 MB 2025-02-14 23:57:47,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 23:57:47,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 23:57:47,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 23:57:47,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:47,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45453.24 MB 2025-02-14 23:57:47,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47342.77 MB 2025-02-14 23:57:47,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 23:57:47,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53953.43 MB 2025-02-14 23:57:47,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53955.53 MB 2025-02-14 23:57:47,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 23:57:47,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48760.20 MB 2025-02-14 23:57:48,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 23:57:48,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 23:57:48,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 23:57:48,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:48,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47342.77 MB 2025-02-14 23:57:48,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49584.63 MB 2025-02-14 23:57:48,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 23:57:48,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53955.53 MB 2025-02-14 23:57:48,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58202.26 MB 2025-02-14 23:57:48,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 23:57:48,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55128.91 MB 2025-02-14 23:57:48,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 23:57:48,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 23:57:48,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 23:57:48,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:48,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45453.24 MB 2025-02-14 23:57:48,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49584.63 MB 2025-02-14 23:57:48,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 23:57:48,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53953.43 MB 2025-02-14 23:57:48,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58202.26 MB 2025-02-14 23:57:48,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4248.83 MB 2025-02-14 23:57:48,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55128.91 MB 2025-02-14 23:57:48,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 23:57:48,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 23:57:48,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 23:57:48,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:48,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51118.17 MB 2025-02-14 23:57:48,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51885.17 MB 2025-02-14 23:57:48,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 23:57:48,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58202.26 MB 2025-02-14 23:57:48,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58619.59 MB 2025-02-14 23:57:48,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 23:57:48,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52592.96 MB 2025-02-14 23:57:48,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 23:57:48,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 23:57:48,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:57:48,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:48,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52298.06 MB 2025-02-14 23:57:48,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52526.78 MB 2025-02-14 23:57:48,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-14 23:57:48,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58619.59 MB 2025-02-14 23:57:48,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58619.59 MB 2025-02-14 23:57:48,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:57:48,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52752.27 MB 2025-02-14 23:57:48,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 23:57:48,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 23:57:48,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.65 seconds 2025-02-14 23:57:48,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:48,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39212.14 MB 2025-02-14 23:57:48,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52727.41 MB 2025-02-14 23:57:48,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13515.27 MB 2025-02-14 23:57:48,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73024.93 MB 2025-02-14 23:57:48,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58619.59 MB 2025-02-14 23:57:48,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14405.34 MB 2025-02-14 23:57:48,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52752.27 MB 2025-02-14 23:57:48,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 23:57:48,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 23:57:48,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 23:57:48,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:48,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52727.41 MB 2025-02-14 23:57:48,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44209.83 MB 2025-02-14 23:57:48,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8517.58 MB 2025-02-14 23:57:48,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58619.59 MB 2025-02-14 23:57:48,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58619.59 MB 2025-02-14 23:57:48,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 23:57:48,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55233.55 MB 2025-02-14 23:57:48,559 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 23:57:48,560 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:57:48,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 23:57:48,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 23:57:48,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 23:57:48,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 23:57:48,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44209.83 MB 2025-02-14 23:57:48,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52630.61 MB 2025-02-14 23:57:48,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 23:57:48,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58619.59 MB 2025-02-14 23:57:48,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66991.42 MB 2025-02-14 23:57:48,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 23:57:48,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52630.61 MB 2025-02-14 23:57:48,721 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 23:57:48,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:57:48,722 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 23:57:48,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:57:48,723 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 23:57:48,728 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 23:57:48,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:57:48,729 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 23:57:48,729 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 23:59:52,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:59:52,568 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 23:59:52,574 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 23:59:52,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:59:52,579 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1088, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 23:59:52,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 23:59:52,580 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1088, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:00:09,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:00:09,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:00:09,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.70 seconds 2025-02-15 00:00:09,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:09,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40898.43 MB 2025-02-15 00:00:09,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44748.80 MB 2025-02-15 00:00:09,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3850.37 MB 2025-02-15 00:00:09,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75363.25 MB 2025-02-15 00:00:09,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48033.17 MB 2025-02-15 00:00:09,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27330.08 MB 2025-02-15 00:00:09,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53768.45 MB 2025-02-15 00:00:09,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:00:09,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:00:09,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 00:00:09,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:09,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44748.80 MB 2025-02-15 00:00:09,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41783.44 MB 2025-02-15 00:00:09,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2965.36 MB 2025-02-15 00:00:09,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48033.17 MB 2025-02-15 00:00:09,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63214.45 MB 2025-02-15 00:00:09,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15181.28 MB 2025-02-15 00:00:09,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55971.26 MB 2025-02-15 00:00:11,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:00:11,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:00:11,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 00:00:11,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41783.44 MB 2025-02-15 00:00:11,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42314.28 MB 2025-02-15 00:00:11,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:00:11,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63214.45 MB 2025-02-15 00:00:11,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47016.05 MB 2025-02-15 00:00:11,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16198.40 MB 2025-02-15 00:00:11,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46292.83 MB 2025-02-15 00:00:11,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:00:11,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:00:11,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:00:11,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42314.28 MB 2025-02-15 00:00:11,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44203.81 MB 2025-02-15 00:00:11,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:00:11,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47016.05 MB 2025-02-15 00:00:11,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48905.58 MB 2025-02-15 00:00:11,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 00:00:11,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45621.24 MB 2025-02-15 00:00:11,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:00:11,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:00:11,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:00:11,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44203.81 MB 2025-02-15 00:00:11,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46445.67 MB 2025-02-15 00:00:11,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:00:11,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48905.58 MB 2025-02-15 00:00:11,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55039.75 MB 2025-02-15 00:00:11,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 00:00:11,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51989.95 MB 2025-02-15 00:00:11,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:00:11,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:00:11,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:00:11,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42314.28 MB 2025-02-15 00:00:11,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46445.67 MB 2025-02-15 00:00:11,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:00:11,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47016.05 MB 2025-02-15 00:00:11,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55039.75 MB 2025-02-15 00:00:11,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-15 00:00:11,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51989.95 MB 2025-02-15 00:00:11,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:00:11,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:00:11,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 00:00:11,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47979.21 MB 2025-02-15 00:00:11,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48746.21 MB 2025-02-15 00:00:11,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:00:11,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55039.75 MB 2025-02-15 00:00:11,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55457.09 MB 2025-02-15 00:00:11,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:00:11,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49454.00 MB 2025-02-15 00:00:11,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:00:11,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:00:11,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:00:11,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49159.10 MB 2025-02-15 00:00:11,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49388.21 MB 2025-02-15 00:00:11,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-15 00:00:11,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55457.09 MB 2025-02-15 00:00:11,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55457.09 MB 2025-02-15 00:00:11,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:00:11,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49587.56 MB 2025-02-15 00:00:11,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:00:11,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:00:11,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.10 seconds 2025-02-15 00:00:11,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37107.75 MB 2025-02-15 00:00:11,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49589.24 MB 2025-02-15 00:00:11,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12481.48 MB 2025-02-15 00:00:11,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75363.25 MB 2025-02-15 00:00:11,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55457.09 MB 2025-02-15 00:00:11,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19906.17 MB 2025-02-15 00:00:11,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49589.24 MB 2025-02-15 00:00:11,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:00:11,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:00:11,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:00:11,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49589.24 MB 2025-02-15 00:00:11,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42111.38 MB 2025-02-15 00:00:11,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7477.86 MB 2025-02-15 00:00:11,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55457.09 MB 2025-02-15 00:00:11,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55457.09 MB 2025-02-15 00:00:11,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:00:11,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52100.29 MB 2025-02-15 00:00:11,970 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-15 00:00:11,971 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:00:11,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:00:11,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:00:11,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:00:11,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:00:11,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42111.38 MB 2025-02-15 00:00:11,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50548.85 MB 2025-02-15 00:00:11,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-15 00:00:11,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55457.09 MB 2025-02-15 00:00:11,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63845.70 MB 2025-02-15 00:00:11,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 00:00:11,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50548.85 MB 2025-02-15 00:00:12,133 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-15 00:00:12,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:00:12,134 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:00:12,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:00:12,135 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:00:12,140 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:00:12,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:00:12,141 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:00:12,141 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:01:19,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:01:19,063 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:01:19,068 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:01:19,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:01:19,072 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2959, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:01:19,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:01:19,073 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2959, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:02:04,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:02:04,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:02:04,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.85 seconds 2025-02-15 00:02:04,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:04,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53937.30 MB 2025-02-15 00:02:04,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 64409.04 MB 2025-02-15 00:02:04,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10471.74 MB 2025-02-15 00:02:04,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 92857.70 MB 2025-02-15 00:02:04,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66932.70 MB 2025-02-15 00:02:04,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25924.99 MB 2025-02-15 00:02:04,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 74880.77 MB 2025-02-15 00:02:05,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:02:05,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:02:05,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 00:02:05,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:05,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 64409.04 MB 2025-02-15 00:02:05,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51510.91 MB 2025-02-15 00:02:05,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12898.13 MB 2025-02-15 00:02:05,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66932.70 MB 2025-02-15 00:02:05,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 91664.42 MB 2025-02-15 00:02:05,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24731.71 MB 2025-02-15 00:02:05,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 95263.64 MB 2025-02-15 00:02:07,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:02:07,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:02:07,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-15 00:02:07,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51510.91 MB 2025-02-15 00:02:07,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52041.75 MB 2025-02-15 00:02:07,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:02:07,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 91664.42 MB 2025-02-15 00:02:07,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54056.19 MB 2025-02-15 00:02:07,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37608.23 MB 2025-02-15 00:02:07,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56020.29 MB 2025-02-15 00:02:07,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:02:07,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:02:07,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:02:07,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52041.75 MB 2025-02-15 00:02:07,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53931.28 MB 2025-02-15 00:02:07,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:02:07,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54056.19 MB 2025-02-15 00:02:07,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57833.16 MB 2025-02-15 00:02:07,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3776.97 MB 2025-02-15 00:02:07,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55348.71 MB 2025-02-15 00:02:07,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:02:07,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:02:07,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:02:07,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53931.28 MB 2025-02-15 00:02:07,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 56173.14 MB 2025-02-15 00:02:07,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:02:07,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57833.16 MB 2025-02-15 00:02:07,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63967.33 MB 2025-02-15 00:02:07,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 00:02:07,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61717.42 MB 2025-02-15 00:02:07,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:02:07,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:02:07,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 00:02:07,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52041.75 MB 2025-02-15 00:02:07,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 56173.14 MB 2025-02-15 00:02:07,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:02:07,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54056.19 MB 2025-02-15 00:02:07,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63967.33 MB 2025-02-15 00:02:07,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9911.14 MB 2025-02-15 00:02:07,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61717.42 MB 2025-02-15 00:02:07,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:02:07,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:02:07,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 00:02:07,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 57706.68 MB 2025-02-15 00:02:07,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 58473.68 MB 2025-02-15 00:02:07,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:02:07,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63967.33 MB 2025-02-15 00:02:07,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64384.66 MB 2025-02-15 00:02:07,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:02:07,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59181.47 MB 2025-02-15 00:02:07,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:02:07,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:02:07,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:02:07,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 58886.57 MB 2025-02-15 00:02:07,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 59114.70 MB 2025-02-15 00:02:07,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-15 00:02:07,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64384.66 MB 2025-02-15 00:02:07,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64384.66 MB 2025-02-15 00:02:07,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:02:07,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59331.34 MB 2025-02-15 00:02:07,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:02:07,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:02:07,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 48.56 seconds 2025-02-15 00:02:07,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43627.19 MB 2025-02-15 00:02:07,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 59315.25 MB 2025-02-15 00:02:07,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15688.07 MB 2025-02-15 00:02:07,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82546.00 MB 2025-02-15 00:02:07,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64384.66 MB 2025-02-15 00:02:07,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18161.34 MB 2025-02-15 00:02:07,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59331.34 MB 2025-02-15 00:02:07,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:02:07,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:02:07,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:02:07,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 59315.25 MB 2025-02-15 00:02:07,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48623.26 MB 2025-02-15 00:02:07,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10691.99 MB 2025-02-15 00:02:07,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64384.66 MB 2025-02-15 00:02:07,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64384.66 MB 2025-02-15 00:02:07,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:02:07,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61820.47 MB 2025-02-15 00:02:07,919 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-15 00:02:07,920 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-15 00:02:07,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:02:07,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:02:07,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:02:07,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:07,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48623.26 MB 2025-02-15 00:02:07,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 57040.38 MB 2025-02-15 00:02:07,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.12 MB 2025-02-15 00:02:07,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64384.66 MB 2025-02-15 00:02:07,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 68568.48 MB 2025-02-15 00:02:07,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-15 00:02:07,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57040.38 MB 2025-02-15 00:02:08,089 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-15 00:02:08,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:08,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:02:08,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:08,091 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:02:08,096 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:02:08,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:08,097 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:02:08,097 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-15 00:02:23,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:23,264 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:02:23,271 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:02:23,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:23,278 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1537, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:02:23,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:23,280 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1537, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:02:47,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:02:47,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:02:47,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.04 seconds 2025-02-15 00:02:47,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:47,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44027.14 MB 2025-02-15 00:02:47,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49467.15 MB 2025-02-15 00:02:47,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5440.01 MB 2025-02-15 00:02:47,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76936.12 MB 2025-02-15 00:02:47,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58902.71 MB 2025-02-15 00:02:47,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18033.41 MB 2025-02-15 00:02:47,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58481.80 MB 2025-02-15 00:02:47,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:02:47,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:02:47,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 00:02:47,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:47,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49467.15 MB 2025-02-15 00:02:47,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44116.60 MB 2025-02-15 00:02:47,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5350.55 MB 2025-02-15 00:02:47,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58902.71 MB 2025-02-15 00:02:47,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70694.99 MB 2025-02-15 00:02:47,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11792.29 MB 2025-02-15 00:02:47,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65167.13 MB 2025-02-15 00:02:49,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:02:49,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:02:49,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 00:02:49,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:49,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44116.60 MB 2025-02-15 00:02:49,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44647.44 MB 2025-02-15 00:02:49,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:02:49,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70694.99 MB 2025-02-15 00:02:49,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49278.88 MB 2025-02-15 00:02:49,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21416.12 MB 2025-02-15 00:02:49,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48627.03 MB 2025-02-15 00:02:49,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:02:49,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:02:49,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:02:49,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:49,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44647.44 MB 2025-02-15 00:02:49,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46536.98 MB 2025-02-15 00:02:49,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:02:49,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49278.88 MB 2025-02-15 00:02:49,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51168.41 MB 2025-02-15 00:02:49,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 00:02:49,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47954.41 MB 2025-02-15 00:02:49,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:02:49,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:02:49,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:02:49,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:49,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46536.98 MB 2025-02-15 00:02:49,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48778.83 MB 2025-02-15 00:02:49,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:02:49,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51168.41 MB 2025-02-15 00:02:49,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56830.72 MB 2025-02-15 00:02:49,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 00:02:49,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54323.11 MB 2025-02-15 00:02:49,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:02:49,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:02:49,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:02:49,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:49,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44647.44 MB 2025-02-15 00:02:49,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48778.83 MB 2025-02-15 00:02:49,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:02:49,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49278.88 MB 2025-02-15 00:02:49,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56830.72 MB 2025-02-15 00:02:49,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7551.84 MB 2025-02-15 00:02:49,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54323.11 MB 2025-02-15 00:02:49,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:02:49,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:02:49,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 00:02:49,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:49,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50312.38 MB 2025-02-15 00:02:49,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51079.38 MB 2025-02-15 00:02:49,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:02:49,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56830.72 MB 2025-02-15 00:02:49,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57248.06 MB 2025-02-15 00:02:49,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:02:49,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51787.17 MB 2025-02-15 00:02:49,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:02:49,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:02:49,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:02:49,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:49,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51492.27 MB 2025-02-15 00:02:49,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51720.82 MB 2025-02-15 00:02:49,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.56 MB 2025-02-15 00:02:49,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57248.06 MB 2025-02-15 00:02:49,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57248.06 MB 2025-02-15 00:02:49,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:02:49,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51940.62 MB 2025-02-15 00:02:49,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:02:49,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:02:49,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.49 seconds 2025-02-15 00:02:49,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:49,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38672.10 MB 2025-02-15 00:02:49,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51921.35 MB 2025-02-15 00:02:49,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13249.25 MB 2025-02-15 00:02:49,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76936.12 MB 2025-02-15 00:02:49,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57248.06 MB 2025-02-15 00:02:49,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19688.06 MB 2025-02-15 00:02:49,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51940.62 MB 2025-02-15 00:02:50,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:02:50,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:02:50,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:02:50,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:50,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51921.35 MB 2025-02-15 00:02:50,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43668.32 MB 2025-02-15 00:02:50,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8253.03 MB 2025-02-15 00:02:50,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57248.06 MB 2025-02-15 00:02:50,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57248.06 MB 2025-02-15 00:02:50,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:02:50,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54426.26 MB 2025-02-15 00:02:50,061 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-15 00:02:50,062 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:02:50,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:02:50,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:02:50,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:02:50,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:02:50,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43668.32 MB 2025-02-15 00:02:50,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52084.92 MB 2025-02-15 00:02:50,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-15 00:02:50,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57248.06 MB 2025-02-15 00:02:50,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65615.69 MB 2025-02-15 00:02:50,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-15 00:02:50,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52084.92 MB 2025-02-15 00:02:50,226 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-15 00:02:50,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:50,227 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:02:50,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:50,228 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:02:50,233 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:02:50,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:50,234 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:02:50,234 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:02:59,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:59,724 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:02:59,729 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:02:59,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:59,732 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:02:59,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:02:59,733 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:03:03,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:03:03,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:03:03,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.68 seconds 2025-02-15 00:03:03,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:03,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34926.72 MB 2025-02-15 00:03:03,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35744.61 MB 2025-02-15 00:03:03,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.89 MB 2025-02-15 00:03:03,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73983.33 MB 2025-02-15 00:03:03,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38543.56 MB 2025-02-15 00:03:03,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35439.77 MB 2025-02-15 00:03:03,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44625.04 MB 2025-02-15 00:03:03,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:03:03,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:03:03,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:03:03,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:03,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35744.61 MB 2025-02-15 00:03:03,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36099.16 MB 2025-02-15 00:03:03,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 354.55 MB 2025-02-15 00:03:03,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38543.56 MB 2025-02-15 00:03:03,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41307.60 MB 2025-02-15 00:03:03,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2764.05 MB 2025-02-15 00:03:03,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38931.56 MB 2025-02-15 00:03:04,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:03:04,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:03:04,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.09 seconds 2025-02-15 00:03:04,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:04,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36099.16 MB 2025-02-15 00:03:04,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36397.76 MB 2025-02-15 00:03:04,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.60 MB 2025-02-15 00:03:04,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41307.60 MB 2025-02-15 00:03:04,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38835.06 MB 2025-02-15 00:03:04,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2472.54 MB 2025-02-15 00:03:04,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40355.82 MB 2025-02-15 00:03:04,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:03:04,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:03:04,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:03:04,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:04,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36397.76 MB 2025-02-15 00:03:04,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37460.36 MB 2025-02-15 00:03:04,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1062.60 MB 2025-02-15 00:03:04,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38835.06 MB 2025-02-15 00:03:04,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39900.41 MB 2025-02-15 00:03:04,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1065.35 MB 2025-02-15 00:03:04,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38257.67 MB 2025-02-15 00:03:04,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:03:04,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:03:04,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 00:03:04,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:04,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37460.36 MB 2025-02-15 00:03:04,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38721.43 MB 2025-02-15 00:03:04,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.07 MB 2025-02-15 00:03:04,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39900.41 MB 2025-02-15 00:03:04,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43362.81 MB 2025-02-15 00:03:04,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3462.40 MB 2025-02-15 00:03:04,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41840.98 MB 2025-02-15 00:03:04,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:03:04,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:03:04,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 00:03:04,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:04,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36397.76 MB 2025-02-15 00:03:04,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38721.43 MB 2025-02-15 00:03:04,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2323.68 MB 2025-02-15 00:03:04,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38835.06 MB 2025-02-15 00:03:04,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43362.81 MB 2025-02-15 00:03:04,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4527.75 MB 2025-02-15 00:03:04,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41840.98 MB 2025-02-15 00:03:04,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:03:04,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:03:04,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 00:03:04,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:04,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39584.05 MB 2025-02-15 00:03:04,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40016.41 MB 2025-02-15 00:03:04,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.36 MB 2025-02-15 00:03:04,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43362.81 MB 2025-02-15 00:03:04,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43597.69 MB 2025-02-15 00:03:04,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 234.88 MB 2025-02-15 00:03:04,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40414.62 MB 2025-02-15 00:03:04,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:03:04,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:03:04,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 00:03:04,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:04,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40248.66 MB 2025-02-15 00:03:04,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40476.97 MB 2025-02-15 00:03:04,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.31 MB 2025-02-15 00:03:04,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43597.69 MB 2025-02-15 00:03:04,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43597.69 MB 2025-02-15 00:03:04,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:03:04,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40554.23 MB 2025-02-15 00:03:04,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:03:04,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:03:04,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.11 seconds 2025-02-15 00:03:04,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:04,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34121.90 MB 2025-02-15 00:03:04,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40678.04 MB 2025-02-15 00:03:04,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6556.15 MB 2025-02-15 00:03:04,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73983.33 MB 2025-02-15 00:03:04,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43597.69 MB 2025-02-15 00:03:04,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30385.64 MB 2025-02-15 00:03:04,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40678.04 MB 2025-02-15 00:03:05,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:03:05,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:03:05,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:03:05,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:05,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35287.03 MB 2025-02-15 00:03:05,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38301.06 MB 2025-02-15 00:03:05,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-15 00:03:05,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43597.69 MB 2025-02-15 00:03:05,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43597.69 MB 2025-02-15 00:03:05,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:03:05,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38602.43 MB 2025-02-15 00:03:05,140 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 00:03:05,140 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 00:03:05,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:03:05,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:03:05,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:03:05,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:05,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38301.06 MB 2025-02-15 00:03:05,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46740.09 MB 2025-02-15 00:03:05,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 00:03:05,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43597.69 MB 2025-02-15 00:03:05,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54087.65 MB 2025-02-15 00:03:05,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 00:03:05,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46740.09 MB 2025-02-15 00:03:05,306 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 00:03:05,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:05,307 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:03:05,308 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:05,308 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:03:05,313 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:03:05,314 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:05,314 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:03:05,314 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 00:03:17,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:17,931 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:03:17,936 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:03:17,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:17,940 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:03:17,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:17,941 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:03:20,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:03:20,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:03:20,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-15 00:03:20,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:20,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34418.04 MB 2025-02-15 00:03:20,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34977.98 MB 2025-02-15 00:03:20,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.94 MB 2025-02-15 00:03:20,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66672.66 MB 2025-02-15 00:03:20,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36781.95 MB 2025-02-15 00:03:20,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29890.71 MB 2025-02-15 00:03:20,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43890.68 MB 2025-02-15 00:03:20,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:03:20,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:03:20,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:03:20,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:20,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34977.98 MB 2025-02-15 00:03:20,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35158.34 MB 2025-02-15 00:03:20,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 180.36 MB 2025-02-15 00:03:20,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36781.95 MB 2025-02-15 00:03:20,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38566.63 MB 2025-02-15 00:03:20,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1784.68 MB 2025-02-15 00:03:20,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37067.40 MB 2025-02-15 00:03:21,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:03:21,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:03:21,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-15 00:03:21,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35158.34 MB 2025-02-15 00:03:21,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35350.77 MB 2025-02-15 00:03:21,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-15 00:03:21,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38566.63 MB 2025-02-15 00:03:21,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37243.32 MB 2025-02-15 00:03:21,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1323.30 MB 2025-02-15 00:03:21,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39330.06 MB 2025-02-15 00:03:21,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:03:21,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:03:21,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-15 00:03:21,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35350.77 MB 2025-02-15 00:03:21,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36035.56 MB 2025-02-15 00:03:21,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-15 00:03:21,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37243.32 MB 2025-02-15 00:03:21,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37933.29 MB 2025-02-15 00:03:21,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 689.96 MB 2025-02-15 00:03:21,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36549.38 MB 2025-02-15 00:03:21,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:03:21,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:03:21,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 00:03:21,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36035.56 MB 2025-02-15 00:03:21,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36848.27 MB 2025-02-15 00:03:21,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-15 00:03:21,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37933.29 MB 2025-02-15 00:03:21,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39996.88 MB 2025-02-15 00:03:21,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2063.60 MB 2025-02-15 00:03:21,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38859.87 MB 2025-02-15 00:03:21,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:03:21,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:03:21,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 00:03:21,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35350.77 MB 2025-02-15 00:03:21,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36848.27 MB 2025-02-15 00:03:21,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-15 00:03:21,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37243.32 MB 2025-02-15 00:03:21,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39996.88 MB 2025-02-15 00:03:21,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2753.56 MB 2025-02-15 00:03:21,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38859.87 MB 2025-02-15 00:03:21,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:03:21,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:03:21,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-15 00:03:21,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37404.18 MB 2025-02-15 00:03:21,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37682.39 MB 2025-02-15 00:03:21,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.21 MB 2025-02-15 00:03:21,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39996.88 MB 2025-02-15 00:03:21,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40147.88 MB 2025-02-15 00:03:21,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-15 00:03:21,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37949.57 MB 2025-02-15 00:03:21,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:03:21,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:03:21,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:03:21,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37832.07 MB 2025-02-15 00:03:21,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38059.53 MB 2025-02-15 00:03:21,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.46 MB 2025-02-15 00:03:21,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40147.88 MB 2025-02-15 00:03:21,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40147.88 MB 2025-02-15 00:03:21,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:03:21,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38069.15 MB 2025-02-15 00:03:21,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:03:21,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:03:21,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.40 seconds 2025-02-15 00:03:21,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33867.56 MB 2025-02-15 00:03:21,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38260.31 MB 2025-02-15 00:03:21,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4392.75 MB 2025-02-15 00:03:21,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66672.66 MB 2025-02-15 00:03:21,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40147.88 MB 2025-02-15 00:03:21,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26524.78 MB 2025-02-15 00:03:21,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38260.31 MB 2025-02-15 00:03:21,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:03:21,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:03:21,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:03:21,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38260.31 MB 2025-02-15 00:03:21,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37663.95 MB 2025-02-15 00:03:21,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -596.36 MB 2025-02-15 00:03:21,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40147.88 MB 2025-02-15 00:03:21,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40550.53 MB 2025-02-15 00:03:21,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-15 00:03:21,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39364.28 MB 2025-02-15 00:03:21,626 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-15 00:03:21,627 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 00:03:21,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:03:21,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:03:21,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:03:21,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:03:21,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37663.95 MB 2025-02-15 00:03:21,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46090.46 MB 2025-02-15 00:03:21,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-15 00:03:21,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40550.53 MB 2025-02-15 00:03:21,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51023.71 MB 2025-02-15 00:03:21,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-15 00:03:21,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46090.46 MB 2025-02-15 00:03:21,788 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-15 00:03:21,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:21,790 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:03:21,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:21,791 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:03:21,795 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:03:21,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:03:21,796 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:03:21,797 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 00:04:40,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:04:40,548 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:04:40,556 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:04:40,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:04:40,563 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:04:40,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:04:40,565 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:04:43,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:04:43,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:04:43,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-15 00:04:43,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:43,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34787.36 MB 2025-02-15 00:04:43,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35534.07 MB 2025-02-15 00:04:43,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.72 MB 2025-02-15 00:04:43,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63589.84 MB 2025-02-15 00:04:43,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38830.87 MB 2025-02-15 00:04:43,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24758.98 MB 2025-02-15 00:04:43,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44485.68 MB 2025-02-15 00:04:43,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:04:43,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:04:43,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:04:43,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:43,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35534.07 MB 2025-02-15 00:04:43,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35821.12 MB 2025-02-15 00:04:43,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 287.05 MB 2025-02-15 00:04:43,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38830.87 MB 2025-02-15 00:04:43,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40238.06 MB 2025-02-15 00:04:43,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1407.19 MB 2025-02-15 00:04:43,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38356.48 MB 2025-02-15 00:04:44,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:04:44,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:04:44,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.96 seconds 2025-02-15 00:04:44,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:44,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35821.12 MB 2025-02-15 00:04:44,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36086.54 MB 2025-02-15 00:04:44,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 265.42 MB 2025-02-15 00:04:44,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40238.06 MB 2025-02-15 00:04:44,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39724.25 MB 2025-02-15 00:04:44,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -513.80 MB 2025-02-15 00:04:44,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40076.74 MB 2025-02-15 00:04:44,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:04:44,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:04:44,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:04:44,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:44,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36086.54 MB 2025-02-15 00:04:44,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37031.08 MB 2025-02-15 00:04:44,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 944.54 MB 2025-02-15 00:04:44,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39724.25 MB 2025-02-15 00:04:44,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39726.35 MB 2025-02-15 00:04:44,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 00:04:44,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37739.80 MB 2025-02-15 00:04:44,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:04:44,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:04:44,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-15 00:04:44,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:44,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37031.08 MB 2025-02-15 00:04:44,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38152.04 MB 2025-02-15 00:04:44,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1120.96 MB 2025-02-15 00:04:44,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39726.35 MB 2025-02-15 00:04:44,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42557.51 MB 2025-02-15 00:04:44,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-15 00:04:44,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40924.15 MB 2025-02-15 00:04:44,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:04:44,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:04:44,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 00:04:44,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:44,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36086.54 MB 2025-02-15 00:04:44,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38152.04 MB 2025-02-15 00:04:44,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2065.50 MB 2025-02-15 00:04:44,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39724.25 MB 2025-02-15 00:04:44,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42557.51 MB 2025-02-15 00:04:44,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-15 00:04:44,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40924.15 MB 2025-02-15 00:04:45,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:04:45,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:04:45,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-15 00:04:45,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:45,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38918.81 MB 2025-02-15 00:04:45,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39302.31 MB 2025-02-15 00:04:45,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 383.50 MB 2025-02-15 00:04:45,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42557.51 MB 2025-02-15 00:04:45,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42765.12 MB 2025-02-15 00:04:45,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-15 00:04:45,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39657.82 MB 2025-02-15 00:04:45,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:04:45,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:04:45,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:04:45,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:45,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39508.76 MB 2025-02-15 00:04:45,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39738.21 MB 2025-02-15 00:04:45,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.45 MB 2025-02-15 00:04:45,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42765.12 MB 2025-02-15 00:04:45,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42765.12 MB 2025-02-15 00:04:45,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:04:45,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39792.19 MB 2025-02-15 00:04:45,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:04:45,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:04:45,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.48 seconds 2025-02-15 00:04:45,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:45,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34052.21 MB 2025-02-15 00:04:45,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39939.02 MB 2025-02-15 00:04:45,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5886.80 MB 2025-02-15 00:04:45,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63589.84 MB 2025-02-15 00:04:45,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42767.22 MB 2025-02-15 00:04:45,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20822.62 MB 2025-02-15 00:04:45,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39939.02 MB 2025-02-15 00:04:45,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:04:45,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:04:45,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 00:04:45,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:45,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35098.58 MB 2025-02-15 00:04:45,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38108.63 MB 2025-02-15 00:04:45,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.05 MB 2025-02-15 00:04:45,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42767.22 MB 2025-02-15 00:04:45,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42767.22 MB 2025-02-15 00:04:45,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:04:45,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38409.59 MB 2025-02-15 00:04:45,335 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-15 00:04:45,335 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:04:45,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:04:45,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:04:45,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:04:45,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:04:45,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38108.63 MB 2025-02-15 00:04:45,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46535.97 MB 2025-02-15 00:04:45,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-15 00:04:45,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42767.22 MB 2025-02-15 00:04:45,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53242.49 MB 2025-02-15 00:04:45,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-15 00:04:45,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46535.97 MB 2025-02-15 00:04:45,501 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-15 00:04:45,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:04:45,503 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:04:45,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:04:45,504 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:04:45,508 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:04:45,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:04:45,509 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:04:45,509 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:06:17,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:06:17,711 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:06:17,716 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:06:17,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:06:17,721 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1572, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:06:17,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:06:17,723 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1572, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:06:41,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:06:41,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:06:41,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.00 seconds 2025-02-15 00:06:41,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:41,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44271.02 MB 2025-02-15 00:06:41,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49834.77 MB 2025-02-15 00:06:41,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5563.74 MB 2025-02-15 00:06:41,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61622.71 MB 2025-02-15 00:06:41,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58160.32 MB 2025-02-15 00:06:41,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3462.40 MB 2025-02-15 00:06:41,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58726.49 MB 2025-02-15 00:06:41,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:06:41,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:06:41,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 00:06:41,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:41,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49834.77 MB 2025-02-15 00:06:41,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44298.56 MB 2025-02-15 00:06:41,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5536.21 MB 2025-02-15 00:06:41,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58160.32 MB 2025-02-15 00:06:41,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70650.95 MB 2025-02-15 00:06:41,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12490.64 MB 2025-02-15 00:06:41,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65272.40 MB 2025-02-15 00:06:43,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:06:43,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:06:43,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 00:06:43,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:43,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44298.56 MB 2025-02-15 00:06:43,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44829.40 MB 2025-02-15 00:06:43,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:06:43,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70650.95 MB 2025-02-15 00:06:43,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52596.57 MB 2025-02-15 00:06:43,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18054.38 MB 2025-02-15 00:06:43,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48807.94 MB 2025-02-15 00:06:43,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:06:43,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:06:43,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:06:43,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:43,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44829.40 MB 2025-02-15 00:06:43,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46718.93 MB 2025-02-15 00:06:43,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:06:43,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52596.57 MB 2025-02-15 00:06:43,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52598.67 MB 2025-02-15 00:06:43,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 00:06:43,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48136.36 MB 2025-02-15 00:06:43,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:06:43,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:06:43,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 00:06:43,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:43,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46718.93 MB 2025-02-15 00:06:43,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48960.79 MB 2025-02-15 00:06:43,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:06:43,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52598.67 MB 2025-02-15 00:06:43,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57789.12 MB 2025-02-15 00:06:43,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 00:06:43,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54505.07 MB 2025-02-15 00:06:43,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:06:43,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:06:43,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:06:43,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:43,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44829.40 MB 2025-02-15 00:06:43,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48960.79 MB 2025-02-15 00:06:43,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:06:43,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52596.57 MB 2025-02-15 00:06:43,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57789.12 MB 2025-02-15 00:06:43,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-15 00:06:43,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54505.07 MB 2025-02-15 00:06:44,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:06:44,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:06:44,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 00:06:44,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:44,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50494.33 MB 2025-02-15 00:06:44,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51261.33 MB 2025-02-15 00:06:44,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:06:44,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57789.12 MB 2025-02-15 00:06:44,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58206.45 MB 2025-02-15 00:06:44,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:06:44,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51969.12 MB 2025-02-15 00:06:44,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:06:44,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:06:44,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:06:44,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:44,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51674.22 MB 2025-02-15 00:06:44,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51902.24 MB 2025-02-15 00:06:44,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.02 MB 2025-02-15 00:06:44,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58206.45 MB 2025-02-15 00:06:44,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58206.45 MB 2025-02-15 00:06:44,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:06:44,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52108.14 MB 2025-02-15 00:06:44,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:06:44,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:06:44,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.42 seconds 2025-02-15 00:06:44,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:44,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38794.05 MB 2025-02-15 00:06:44,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52103.28 MB 2025-02-15 00:06:44,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13309.24 MB 2025-02-15 00:06:44,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61622.71 MB 2025-02-15 00:06:44,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58206.45 MB 2025-02-15 00:06:44,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3416.26 MB 2025-02-15 00:06:44,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52108.14 MB 2025-02-15 00:06:44,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:06:44,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:06:44,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:06:44,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:44,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52103.28 MB 2025-02-15 00:06:44,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43798.06 MB 2025-02-15 00:06:44,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8305.23 MB 2025-02-15 00:06:44,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58206.45 MB 2025-02-15 00:06:44,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58206.45 MB 2025-02-15 00:06:44,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:06:44,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54614.64 MB 2025-02-15 00:06:44,435 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-15 00:06:44,435 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:06:44,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:06:44,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:06:44,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:06:44,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:06:44,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43798.06 MB 2025-02-15 00:06:44,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52237.43 MB 2025-02-15 00:06:44,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.37 MB 2025-02-15 00:06:44,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58206.45 MB 2025-02-15 00:06:44,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66595.06 MB 2025-02-15 00:06:44,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-15 00:06:44,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52237.43 MB 2025-02-15 00:06:44,597 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-15 00:06:44,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:06:44,598 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:06:44,599 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:06:44,599 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:06:44,604 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:06:44,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:06:44,605 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:06:44,605 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:07:39,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:07:39,443 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:07:39,448 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:07:39,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:07:39,451 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2012, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:07:39,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:07:39,452 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2012, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:08:10,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:08:10,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:08:10,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.05 seconds 2025-02-15 00:08:10,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:10,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47337.01 MB 2025-02-15 00:08:10,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54457.37 MB 2025-02-15 00:08:10,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7120.36 MB 2025-02-15 00:08:10,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74983.67 MB 2025-02-15 00:08:10,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59718.50 MB 2025-02-15 00:08:10,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15265.17 MB 2025-02-15 00:08:10,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 63378.50 MB 2025-02-15 00:08:10,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:08:10,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:08:10,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-15 00:08:10,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:10,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54457.37 MB 2025-02-15 00:08:10,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46585.98 MB 2025-02-15 00:08:10,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7871.39 MB 2025-02-15 00:08:10,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59718.50 MB 2025-02-15 00:08:10,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 85070.97 MB 2025-02-15 00:08:10,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25352.47 MB 2025-02-15 00:08:10,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 74981.68 MB 2025-02-15 00:08:12,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:08:12,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:08:12,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 00:08:12,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:12,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46585.98 MB 2025-02-15 00:08:12,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47116.82 MB 2025-02-15 00:08:12,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:08:12,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 85070.97 MB 2025-02-15 00:08:12,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49822.04 MB 2025-02-15 00:08:12,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35248.93 MB 2025-02-15 00:08:12,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51096.41 MB 2025-02-15 00:08:12,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:08:12,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:08:12,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:08:12,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:12,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47116.82 MB 2025-02-15 00:08:12,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49006.35 MB 2025-02-15 00:08:12,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:08:12,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49822.04 MB 2025-02-15 00:08:12,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53127.15 MB 2025-02-15 00:08:12,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-15 00:08:12,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50423.78 MB 2025-02-15 00:08:12,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:08:12,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:08:12,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:08:12,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:12,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49006.35 MB 2025-02-15 00:08:12,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51248.21 MB 2025-02-15 00:08:12,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:08:12,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53127.15 MB 2025-02-15 00:08:12,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59261.32 MB 2025-02-15 00:08:12,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 00:08:12,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56792.49 MB 2025-02-15 00:08:12,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:08:12,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:08:12,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:08:12,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:12,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47116.82 MB 2025-02-15 00:08:12,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51248.21 MB 2025-02-15 00:08:12,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:08:12,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49822.04 MB 2025-02-15 00:08:12,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59261.32 MB 2025-02-15 00:08:12,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-15 00:08:12,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56792.49 MB 2025-02-15 00:08:13,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:08:13,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:08:13,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 00:08:13,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:13,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52781.75 MB 2025-02-15 00:08:13,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53548.75 MB 2025-02-15 00:08:13,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:08:13,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59261.32 MB 2025-02-15 00:08:13,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59678.65 MB 2025-02-15 00:08:13,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:08:13,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54256.54 MB 2025-02-15 00:08:13,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:08:13,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:08:13,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:08:13,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:13,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53961.64 MB 2025-02-15 00:08:13,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54190.43 MB 2025-02-15 00:08:13,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-15 00:08:13,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59678.65 MB 2025-02-15 00:08:13,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59678.65 MB 2025-02-15 00:08:13,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:08:13,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54409.35 MB 2025-02-15 00:08:13,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:08:13,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:08:13,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.61 seconds 2025-02-15 00:08:13,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:13,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40327.04 MB 2025-02-15 00:08:13,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54390.89 MB 2025-02-15 00:08:13,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14063.85 MB 2025-02-15 00:08:13,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74983.67 MB 2025-02-15 00:08:13,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59678.65 MB 2025-02-15 00:08:13,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15305.02 MB 2025-02-15 00:08:13,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54409.35 MB 2025-02-15 00:08:13,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:08:13,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:08:13,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:08:13,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:13,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 54390.89 MB 2025-02-15 00:08:13,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45322.15 MB 2025-02-15 00:08:13,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9068.74 MB 2025-02-15 00:08:13,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59678.65 MB 2025-02-15 00:08:13,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59678.65 MB 2025-02-15 00:08:13,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:08:13,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56894.88 MB 2025-02-15 00:08:13,350 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-15 00:08:13,350 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:08:13,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:08:13,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:08:13,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:08:13,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:13,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45322.15 MB 2025-02-15 00:08:13,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53735.10 MB 2025-02-15 00:08:13,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.95 MB 2025-02-15 00:08:13,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59678.65 MB 2025-02-15 00:08:13,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63860.38 MB 2025-02-15 00:08:13,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-15 00:08:13,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53735.10 MB 2025-02-15 00:08:13,511 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-15 00:08:13,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:13,513 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:08:13,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:13,513 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:08:13,518 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:08:13,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:13,519 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:08:13,519 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:08:32,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:32,112 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:08:32,117 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:08:32,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:32,120 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1581, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:08:32,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:32,121 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1581, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:08:56,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:08:56,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:08:56,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.64 seconds 2025-02-15 00:08:56,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:56,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44333.73 MB 2025-02-15 00:08:56,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49928.94 MB 2025-02-15 00:08:56,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5595.20 MB 2025-02-15 00:08:56,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72223.82 MB 2025-02-15 00:08:56,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58166.61 MB 2025-02-15 00:08:56,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14057.21 MB 2025-02-15 00:08:56,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58789.20 MB 2025-02-15 00:08:56,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:08:56,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:08:56,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 00:08:56,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:56,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49928.94 MB 2025-02-15 00:08:56,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44345.34 MB 2025-02-15 00:08:56,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5583.59 MB 2025-02-15 00:08:56,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58166.61 MB 2025-02-15 00:08:56,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69357.01 MB 2025-02-15 00:08:56,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11190.40 MB 2025-02-15 00:08:56,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64372.41 MB 2025-02-15 00:08:58,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:08:58,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:08:58,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 00:08:58,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:58,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44345.34 MB 2025-02-15 00:08:58,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44876.19 MB 2025-02-15 00:08:58,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:08:58,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69357.01 MB 2025-02-15 00:08:58,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52571.41 MB 2025-02-15 00:08:58,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16785.60 MB 2025-02-15 00:08:58,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48854.73 MB 2025-02-15 00:08:58,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:08:58,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:08:58,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:08:58,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:58,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44876.19 MB 2025-02-15 00:08:58,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46765.72 MB 2025-02-15 00:08:58,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:08:58,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52571.41 MB 2025-02-15 00:08:58,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52573.50 MB 2025-02-15 00:08:58,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 00:08:58,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48183.15 MB 2025-02-15 00:08:59,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:08:59,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:08:59,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:08:59,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:59,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46765.72 MB 2025-02-15 00:08:59,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49007.58 MB 2025-02-15 00:08:59,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:08:59,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52573.50 MB 2025-02-15 00:08:59,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58235.81 MB 2025-02-15 00:08:59,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 00:08:59,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54551.86 MB 2025-02-15 00:08:59,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:08:59,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:08:59,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:08:59,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:59,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44876.19 MB 2025-02-15 00:08:59,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49007.58 MB 2025-02-15 00:08:59,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:08:59,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52571.41 MB 2025-02-15 00:08:59,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58235.81 MB 2025-02-15 00:08:59,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-15 00:08:59,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54551.86 MB 2025-02-15 00:08:59,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:08:59,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:08:59,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 00:08:59,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:59,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50541.12 MB 2025-02-15 00:08:59,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51308.12 MB 2025-02-15 00:08:59,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:08:59,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58235.81 MB 2025-02-15 00:08:59,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58653.15 MB 2025-02-15 00:08:59,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:08:59,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52015.91 MB 2025-02-15 00:08:59,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:08:59,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:08:59,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:08:59,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:59,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51721.01 MB 2025-02-15 00:08:59,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51949.93 MB 2025-02-15 00:08:59,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.92 MB 2025-02-15 00:08:59,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58653.15 MB 2025-02-15 00:08:59,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58653.15 MB 2025-02-15 00:08:59,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:08:59,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52180.30 MB 2025-02-15 00:08:59,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:08:59,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:08:59,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.08 seconds 2025-02-15 00:08:59,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:59,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38825.40 MB 2025-02-15 00:08:59,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52150.69 MB 2025-02-15 00:08:59,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13325.28 MB 2025-02-15 00:08:59,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72223.82 MB 2025-02-15 00:08:59,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58653.15 MB 2025-02-15 00:08:59,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13570.67 MB 2025-02-15 00:08:59,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52180.30 MB 2025-02-15 00:08:59,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:08:59,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:08:59,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:08:59,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:59,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52150.69 MB 2025-02-15 00:08:59,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43824.94 MB 2025-02-15 00:08:59,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8325.75 MB 2025-02-15 00:08:59,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58653.15 MB 2025-02-15 00:08:59,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58653.15 MB 2025-02-15 00:08:59,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:08:59,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54658.36 MB 2025-02-15 00:08:59,498 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-15 00:08:59,498 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:08:59,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:08:59,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:08:59,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:08:59,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:08:59,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43824.94 MB 2025-02-15 00:08:59,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52250.92 MB 2025-02-15 00:08:59,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8425.98 MB 2025-02-15 00:08:59,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58653.15 MB 2025-02-15 00:08:59,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62841.16 MB 2025-02-15 00:08:59,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-15 00:08:59,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52250.92 MB 2025-02-15 00:08:59,660 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-15 00:08:59,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:59,661 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:08:59,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:59,662 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:08:59,667 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:08:59,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:08:59,668 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:08:59,668 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:09:54,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:09:54,702 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:09:54,708 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:09:54,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:09:54,713 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 363, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:09:54,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:09:54,714 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 363, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:10:00,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:10:00,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:10:00,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.61 seconds 2025-02-15 00:10:00,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:00,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35846.52 MB 2025-02-15 00:10:00,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37132.07 MB 2025-02-15 00:10:00,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1285.55 MB 2025-02-15 00:10:00,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71217.18 MB 2025-02-15 00:10:00,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39952.84 MB 2025-02-15 00:10:00,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31264.34 MB 2025-02-15 00:10:00,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45998.63 MB 2025-02-15 00:10:00,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:10:00,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:10:00,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:10:00,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:00,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37132.07 MB 2025-02-15 00:10:00,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37754.86 MB 2025-02-15 00:10:00,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 622.79 MB 2025-02-15 00:10:00,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39952.84 MB 2025-02-15 00:10:00,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45212.50 MB 2025-02-15 00:10:00,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5259.66 MB 2025-02-15 00:10:00,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42272.45 MB 2025-02-15 00:10:02,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:10:02,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:10:02,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.74 seconds 2025-02-15 00:10:02,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37754.86 MB 2025-02-15 00:10:02,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38236.60 MB 2025-02-15 00:10:02,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.74 MB 2025-02-15 00:10:02,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45212.50 MB 2025-02-15 00:10:02,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41219.52 MB 2025-02-15 00:10:02,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3992.98 MB 2025-02-15 00:10:02,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42181.39 MB 2025-02-15 00:10:02,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:10:02,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:10:02,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:10:02,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38236.60 MB 2025-02-15 00:10:02,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39950.95 MB 2025-02-15 00:10:02,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1714.36 MB 2025-02-15 00:10:02,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41219.52 MB 2025-02-15 00:10:02,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44654.66 MB 2025-02-15 00:10:02,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3435.13 MB 2025-02-15 00:10:02,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41237.27 MB 2025-02-15 00:10:02,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:10:02,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:10:02,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-15 00:10:02,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39950.95 MB 2025-02-15 00:10:02,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41985.44 MB 2025-02-15 00:10:02,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2034.49 MB 2025-02-15 00:10:02,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44654.66 MB 2025-02-15 00:10:02,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50230.98 MB 2025-02-15 00:10:02,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5576.33 MB 2025-02-15 00:10:02,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47017.03 MB 2025-02-15 00:10:02,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:10:02,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:10:02,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:10:02,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38236.60 MB 2025-02-15 00:10:02,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41985.44 MB 2025-02-15 00:10:02,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3748.85 MB 2025-02-15 00:10:02,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41219.52 MB 2025-02-15 00:10:02,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50230.98 MB 2025-02-15 00:10:02,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9011.46 MB 2025-02-15 00:10:02,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47017.03 MB 2025-02-15 00:10:02,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:10:02,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:10:02,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-15 00:10:02,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43377.29 MB 2025-02-15 00:10:02,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44073.50 MB 2025-02-15 00:10:02,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 696.21 MB 2025-02-15 00:10:02,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50230.98 MB 2025-02-15 00:10:02,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50610.57 MB 2025-02-15 00:10:02,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 379.58 MB 2025-02-15 00:10:02,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44715.82 MB 2025-02-15 00:10:02,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:10:02,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:10:02,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:10:02,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44448.20 MB 2025-02-15 00:10:02,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44660.00 MB 2025-02-15 00:10:02,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.80 MB 2025-02-15 00:10:02,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50610.57 MB 2025-02-15 00:10:02,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50610.57 MB 2025-02-15 00:10:02,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:10:02,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44823.57 MB 2025-02-15 00:10:02,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:10:02,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:10:02,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.76 seconds 2025-02-15 00:10:02,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34581.79 MB 2025-02-15 00:10:02,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44861.08 MB 2025-02-15 00:10:02,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10279.28 MB 2025-02-15 00:10:02,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71217.18 MB 2025-02-15 00:10:02,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50610.57 MB 2025-02-15 00:10:02,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20606.62 MB 2025-02-15 00:10:02,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44861.08 MB 2025-02-15 00:10:02,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:10:02,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:10:02,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:10:02,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44861.08 MB 2025-02-15 00:10:02,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39411.49 MB 2025-02-15 00:10:02,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5449.59 MB 2025-02-15 00:10:02,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50610.57 MB 2025-02-15 00:10:02,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50610.57 MB 2025-02-15 00:10:02,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:10:02,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47975.54 MB 2025-02-15 00:10:02,764 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 00:10:02,764 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-15 00:10:02,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:10:02,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:10:02,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:10:02,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:10:02,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39411.49 MB 2025-02-15 00:10:02,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47850.51 MB 2025-02-15 00:10:02,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-15 00:10:02,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50610.57 MB 2025-02-15 00:10:02,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61100.52 MB 2025-02-15 00:10:02,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-15 00:10:02,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47850.51 MB 2025-02-15 00:10:02,926 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 00:10:02,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:10:02,927 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:10:02,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:10:02,928 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:10:02,933 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:10:02,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:10:02,934 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:10:02,934 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-15 00:11:52,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:11:52,847 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:11:52,852 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:11:52,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:11:52,857 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1142, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:11:52,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:11:52,858 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1142, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:12:10,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:12:10,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:12:10,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.42 seconds 2025-02-15 00:12:10,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:10,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41274.71 MB 2025-02-15 00:12:10,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45316.19 MB 2025-02-15 00:12:10,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4041.47 MB 2025-02-15 00:12:10,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73685.53 MB 2025-02-15 00:12:10,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50367.30 MB 2025-02-15 00:12:10,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23318.23 MB 2025-02-15 00:12:10,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54143.93 MB 2025-02-15 00:12:10,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:12:10,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:12:10,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 00:12:10,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:10,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45316.19 MB 2025-02-15 00:12:10,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42064.17 MB 2025-02-15 00:12:10,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3252.02 MB 2025-02-15 00:12:10,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50367.30 MB 2025-02-15 00:12:10,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63996.69 MB 2025-02-15 00:12:10,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13629.39 MB 2025-02-15 00:12:10,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57550.59 MB 2025-02-15 00:12:12,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:12:12,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:12:12,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-15 00:12:12,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42064.17 MB 2025-02-15 00:12:12,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42595.01 MB 2025-02-15 00:12:12,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:12:12,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63996.69 MB 2025-02-15 00:12:12,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48448.41 MB 2025-02-15 00:12:12,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15548.28 MB 2025-02-15 00:12:12,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46573.56 MB 2025-02-15 00:12:12,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:12:12,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:12:12,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:12:12,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42595.01 MB 2025-02-15 00:12:12,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44484.54 MB 2025-02-15 00:12:12,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:12:12,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48448.41 MB 2025-02-15 00:12:12,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49394.22 MB 2025-02-15 00:12:12,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 945.82 MB 2025-02-15 00:12:12,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45901.97 MB 2025-02-15 00:12:12,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:12:12,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:12:12,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:12:12,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44484.54 MB 2025-02-15 00:12:12,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46726.40 MB 2025-02-15 00:12:12,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:12:12,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49394.22 MB 2025-02-15 00:12:12,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55056.53 MB 2025-02-15 00:12:12,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 00:12:12,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52270.68 MB 2025-02-15 00:12:12,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:12:12,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:12:12,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 00:12:12,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42595.01 MB 2025-02-15 00:12:12,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46726.40 MB 2025-02-15 00:12:12,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:12:12,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48448.41 MB 2025-02-15 00:12:12,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55056.53 MB 2025-02-15 00:12:12,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 00:12:12,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52270.68 MB 2025-02-15 00:12:12,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:12:12,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:12:12,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 00:12:12,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48259.94 MB 2025-02-15 00:12:12,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49026.94 MB 2025-02-15 00:12:12,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:12:12,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55056.53 MB 2025-02-15 00:12:12,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55473.86 MB 2025-02-15 00:12:12,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:12:12,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49734.73 MB 2025-02-15 00:12:12,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:12:12,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:12:12,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:12:12,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49439.83 MB 2025-02-15 00:12:12,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49667.52 MB 2025-02-15 00:12:12,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.68 MB 2025-02-15 00:12:12,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55473.86 MB 2025-02-15 00:12:12,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55473.86 MB 2025-02-15 00:12:12,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:12:12,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49904.83 MB 2025-02-15 00:12:12,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:12:12,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:12:12,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.83 seconds 2025-02-15 00:12:12,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37295.89 MB 2025-02-15 00:12:12,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49868.20 MB 2025-02-15 00:12:12,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12572.30 MB 2025-02-15 00:12:12,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73685.53 MB 2025-02-15 00:12:12,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55473.86 MB 2025-02-15 00:12:12,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18211.67 MB 2025-02-15 00:12:12,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49904.83 MB 2025-02-15 00:12:12,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:12:12,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:12:12,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:12:12,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49868.20 MB 2025-02-15 00:12:12,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42294.32 MB 2025-02-15 00:12:12,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7573.87 MB 2025-02-15 00:12:12,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55473.86 MB 2025-02-15 00:12:12,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55473.86 MB 2025-02-15 00:12:12,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:12:12,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52374.95 MB 2025-02-15 00:12:12,978 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-15 00:12:12,978 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 00:12:12,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:12:12,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:12:12,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:12:12,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:12:12,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42294.32 MB 2025-02-15 00:12:12,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50716.65 MB 2025-02-15 00:12:12,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-15 00:12:12,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55473.86 MB 2025-02-15 00:12:12,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63847.79 MB 2025-02-15 00:12:12,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-15 00:12:12,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50716.65 MB 2025-02-15 00:12:13,143 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-15 00:12:13,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:12:13,145 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:12:13,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:12:13,146 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:12:13,150 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:12:13,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:12:13,151 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:12:13,151 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 00:12:34,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:12:34,108 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:12:34,112 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:12:34,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:12:34,116 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2619, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:12:34,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:12:34,117 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2619, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:13:14,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:13:14,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:13:14,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.72 seconds 2025-02-15 00:13:14,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:14,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51567.60 MB 2025-02-15 00:13:14,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 60837.01 MB 2025-02-15 00:13:14,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9269.41 MB 2025-02-15 00:13:14,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 94661.25 MB 2025-02-15 00:13:14,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63359.16 MB 2025-02-15 00:13:14,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31302.09 MB 2025-02-15 00:13:14,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 70105.51 MB 2025-02-15 00:13:15,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:13:15,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:13:15,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:13:15,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:15,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 60837.01 MB 2025-02-15 00:13:15,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49743.09 MB 2025-02-15 00:13:15,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11093.92 MB 2025-02-15 00:13:15,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63359.16 MB 2025-02-15 00:13:15,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 85435.88 MB 2025-02-15 00:13:15,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22076.72 MB 2025-02-15 00:13:15,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 87687.58 MB 2025-02-15 00:13:17,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:13:17,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:13:17,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-15 00:13:17,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49743.09 MB 2025-02-15 00:13:17,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50273.93 MB 2025-02-15 00:13:17,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:13:17,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 85435.88 MB 2025-02-15 00:13:17,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52288.29 MB 2025-02-15 00:13:17,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33147.58 MB 2025-02-15 00:13:17,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54252.48 MB 2025-02-15 00:13:17,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:13:17,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:13:17,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:13:17,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50273.93 MB 2025-02-15 00:13:17,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52163.46 MB 2025-02-15 00:13:17,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:13:17,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52288.29 MB 2025-02-15 00:13:17,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56065.26 MB 2025-02-15 00:13:17,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3776.97 MB 2025-02-15 00:13:17,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53580.89 MB 2025-02-15 00:13:17,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:13:17,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:13:17,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:13:17,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52163.46 MB 2025-02-15 00:13:17,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54405.32 MB 2025-02-15 00:13:17,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:13:17,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56065.26 MB 2025-02-15 00:13:17,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62199.43 MB 2025-02-15 00:13:17,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 00:13:17,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59949.60 MB 2025-02-15 00:13:17,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:13:17,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:13:17,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:13:17,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50273.93 MB 2025-02-15 00:13:17,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 54405.32 MB 2025-02-15 00:13:17,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:13:17,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52288.29 MB 2025-02-15 00:13:17,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62199.43 MB 2025-02-15 00:13:17,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9911.14 MB 2025-02-15 00:13:17,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59949.60 MB 2025-02-15 00:13:17,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:13:17,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:13:17,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 00:13:17,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 55938.86 MB 2025-02-15 00:13:17,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 56705.86 MB 2025-02-15 00:13:17,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:13:17,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62199.43 MB 2025-02-15 00:13:17,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62616.76 MB 2025-02-15 00:13:17,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:13:17,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57413.65 MB 2025-02-15 00:13:17,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:13:17,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:13:17,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:13:17,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 57118.75 MB 2025-02-15 00:13:17,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 57347.36 MB 2025-02-15 00:13:17,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.60 MB 2025-02-15 00:13:17,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62616.76 MB 2025-02-15 00:13:17,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62616.76 MB 2025-02-15 00:13:17,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:13:17,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57562.95 MB 2025-02-15 00:13:17,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:13:17,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:13:17,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.31 seconds 2025-02-15 00:13:17,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42442.34 MB 2025-02-15 00:13:17,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 57548.43 MB 2025-02-15 00:13:17,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15106.09 MB 2025-02-15 00:13:17,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 85534.44 MB 2025-02-15 00:13:17,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62616.76 MB 2025-02-15 00:13:17,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22917.68 MB 2025-02-15 00:13:17,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57562.95 MB 2025-02-15 00:13:17,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:13:17,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:13:17,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:13:17,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 57548.43 MB 2025-02-15 00:13:17,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47446.17 MB 2025-02-15 00:13:17,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10102.26 MB 2025-02-15 00:13:17,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62616.76 MB 2025-02-15 00:13:17,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62616.76 MB 2025-02-15 00:13:17,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:13:17,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60060.10 MB 2025-02-15 00:13:17,722 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-15 00:13:17,722 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:13:17,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:13:17,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:13:17,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:13:17,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:13:17,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47446.17 MB 2025-02-15 00:13:17,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 55884.86 MB 2025-02-15 00:13:17,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-15 00:13:17,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62616.76 MB 2025-02-15 00:13:17,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66813.17 MB 2025-02-15 00:13:17,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-15 00:13:17,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55884.86 MB 2025-02-15 00:13:17,891 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-15 00:13:17,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:13:17,892 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:13:17,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:13:17,893 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:13:17,898 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:13:17,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:13:17,899 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:13:17,899 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:14:11,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:14:11,143 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:14:11,151 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:14:11,160 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:14:11,160 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 427, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:14:11,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:14:11,162 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 427, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:14:17,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:14:17,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:14:17,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.70 seconds 2025-02-15 00:14:17,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:17,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36292.48 MB 2025-02-15 00:14:17,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37804.53 MB 2025-02-15 00:14:17,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1512.05 MB 2025-02-15 00:14:17,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75201.77 MB 2025-02-15 00:14:17,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41123.05 MB 2025-02-15 00:14:17,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34078.72 MB 2025-02-15 00:14:17,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46671.08 MB 2025-02-15 00:14:17,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:14:17,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:14:17,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-15 00:14:17,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:17,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37804.53 MB 2025-02-15 00:14:17,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38347.11 MB 2025-02-15 00:14:17,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 542.58 MB 2025-02-15 00:14:17,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41123.05 MB 2025-02-15 00:14:17,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48530.19 MB 2025-02-15 00:14:17,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7407.14 MB 2025-02-15 00:14:17,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45069.04 MB 2025-02-15 00:14:19,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:14:19,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:14:19,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 00:14:19,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:19,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38347.11 MB 2025-02-15 00:14:19,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38877.95 MB 2025-02-15 00:14:19,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:14:19,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48530.19 MB 2025-02-15 00:14:19,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42257.61 MB 2025-02-15 00:14:19,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6272.58 MB 2025-02-15 00:14:19,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42857.53 MB 2025-02-15 00:14:19,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:14:19,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:14:19,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:14:19,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:19,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38877.95 MB 2025-02-15 00:14:19,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40767.48 MB 2025-02-15 00:14:19,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:14:19,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42257.61 MB 2025-02-15 00:14:19,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45562.72 MB 2025-02-15 00:14:19,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-15 00:14:19,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42184.91 MB 2025-02-15 00:14:20,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:14:20,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:14:20,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:14:20,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:20,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40767.48 MB 2025-02-15 00:14:20,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43009.34 MB 2025-02-15 00:14:20,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:14:20,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45562.72 MB 2025-02-15 00:14:20,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51696.89 MB 2025-02-15 00:14:20,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-15 00:14:20,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48553.62 MB 2025-02-15 00:14:20,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:14:20,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:14:20,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:14:20,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:20,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38877.95 MB 2025-02-15 00:14:20,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43009.34 MB 2025-02-15 00:14:20,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:14:20,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42257.61 MB 2025-02-15 00:14:20,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51696.89 MB 2025-02-15 00:14:20,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-15 00:14:20,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48553.62 MB 2025-02-15 00:14:20,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:14:20,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:14:20,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 00:14:20,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:20,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44542.88 MB 2025-02-15 00:14:20,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45309.88 MB 2025-02-15 00:14:20,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:14:20,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51696.89 MB 2025-02-15 00:14:20,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52114.23 MB 2025-02-15 00:14:20,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:14:20,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46017.67 MB 2025-02-15 00:14:20,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:14:20,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:14:20,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:14:20,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:20,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45722.77 MB 2025-02-15 00:14:20,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45951.44 MB 2025-02-15 00:14:20,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-15 00:14:20,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52114.23 MB 2025-02-15 00:14:20,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52114.23 MB 2025-02-15 00:14:20,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:14:20,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46153.56 MB 2025-02-15 00:14:20,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:14:20,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:14:20,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.09 seconds 2025-02-15 00:14:20,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:20,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34804.78 MB 2025-02-15 00:14:20,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46152.02 MB 2025-02-15 00:14:20,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11347.24 MB 2025-02-15 00:14:20,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75201.77 MB 2025-02-15 00:14:20,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52114.23 MB 2025-02-15 00:14:20,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23087.55 MB 2025-02-15 00:14:20,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46153.56 MB 2025-02-15 00:14:20,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:14:20,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:14:20,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:14:20,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:20,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46152.02 MB 2025-02-15 00:14:20,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39801.73 MB 2025-02-15 00:14:20,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6350.29 MB 2025-02-15 00:14:20,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52114.23 MB 2025-02-15 00:14:20,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52114.23 MB 2025-02-15 00:14:20,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:14:20,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48657.54 MB 2025-02-15 00:14:20,546 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 00:14:20,547 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:14:20,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:14:20,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:14:20,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:14:20,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:14:20,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39801.73 MB 2025-02-15 00:14:20,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48219.89 MB 2025-02-15 00:14:20,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-15 00:14:20,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52114.23 MB 2025-02-15 00:14:20,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62576.92 MB 2025-02-15 00:14:20,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-15 00:14:20,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48219.89 MB 2025-02-15 00:14:20,716 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 00:14:20,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:14:20,717 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:14:20,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:14:20,718 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:14:20,723 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:14:20,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:14:20,724 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:14:20,724 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:15:28,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:15:28,378 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:15:28,383 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:15:28,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:15:28,387 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1104, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:15:28,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:15:28,388 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1104, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:15:45,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:15:45,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:15:45,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.93 seconds 2025-02-15 00:15:45,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:45,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41009.92 MB 2025-02-15 00:15:45,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44916.92 MB 2025-02-15 00:15:45,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3906.99 MB 2025-02-15 00:15:45,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75130.47 MB 2025-02-15 00:15:45,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50203.72 MB 2025-02-15 00:15:45,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24926.75 MB 2025-02-15 00:15:45,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53879.14 MB 2025-02-15 00:15:45,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:15:45,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:15:45,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 00:15:45,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:45,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44916.92 MB 2025-02-15 00:15:45,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41866.62 MB 2025-02-15 00:15:45,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3050.30 MB 2025-02-15 00:15:45,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50203.72 MB 2025-02-15 00:15:45,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61570.29 MB 2025-02-15 00:15:45,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11366.56 MB 2025-02-15 00:15:45,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56321.64 MB 2025-02-15 00:15:47,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:15:47,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:15:47,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 00:15:47,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:47,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41866.62 MB 2025-02-15 00:15:47,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42397.46 MB 2025-02-15 00:15:47,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:15:47,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61570.29 MB 2025-02-15 00:15:47,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48421.14 MB 2025-02-15 00:15:47,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13149.14 MB 2025-02-15 00:15:47,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46376.01 MB 2025-02-15 00:15:47,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:15:47,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:15:47,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:15:47,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:47,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42397.46 MB 2025-02-15 00:15:47,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44286.99 MB 2025-02-15 00:15:47,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:15:47,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48421.14 MB 2025-02-15 00:15:47,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49366.96 MB 2025-02-15 00:15:47,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 945.82 MB 2025-02-15 00:15:47,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45704.42 MB 2025-02-15 00:15:47,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:15:47,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:15:47,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:15:47,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:47,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44286.99 MB 2025-02-15 00:15:47,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46528.85 MB 2025-02-15 00:15:47,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:15:47,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49366.96 MB 2025-02-15 00:15:47,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55029.27 MB 2025-02-15 00:15:47,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-15 00:15:47,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52073.13 MB 2025-02-15 00:15:47,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:15:47,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:15:47,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-15 00:15:47,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:47,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42397.46 MB 2025-02-15 00:15:47,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46528.85 MB 2025-02-15 00:15:47,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:15:47,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48421.14 MB 2025-02-15 00:15:47,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55029.27 MB 2025-02-15 00:15:47,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-15 00:15:47,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52073.13 MB 2025-02-15 00:15:47,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:15:47,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:15:47,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-15 00:15:47,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:47,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48062.39 MB 2025-02-15 00:15:47,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48829.39 MB 2025-02-15 00:15:47,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:15:47,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55029.27 MB 2025-02-15 00:15:47,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55446.60 MB 2025-02-15 00:15:47,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:15:47,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49537.18 MB 2025-02-15 00:15:47,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:15:47,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:15:47,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:15:47,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:47,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49242.28 MB 2025-02-15 00:15:47,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49470.42 MB 2025-02-15 00:15:47,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.14 MB 2025-02-15 00:15:47,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55446.60 MB 2025-02-15 00:15:47,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55446.60 MB 2025-02-15 00:15:47,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:15:47,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49677.55 MB 2025-02-15 00:15:47,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:15:47,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:15:47,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.38 seconds 2025-02-15 00:15:47,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:47,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37163.50 MB 2025-02-15 00:15:47,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49671.37 MB 2025-02-15 00:15:47,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12507.87 MB 2025-02-15 00:15:47,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75130.47 MB 2025-02-15 00:15:47,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55446.60 MB 2025-02-15 00:15:47,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19683.87 MB 2025-02-15 00:15:47,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49677.55 MB 2025-02-15 00:15:48,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:15:48,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:15:48,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:15:48,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:48,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49671.37 MB 2025-02-15 00:15:48,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42165.98 MB 2025-02-15 00:15:48,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7505.39 MB 2025-02-15 00:15:48,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55446.60 MB 2025-02-15 00:15:48,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55446.60 MB 2025-02-15 00:15:48,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:15:48,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52181.50 MB 2025-02-15 00:15:48,059 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-15 00:15:48,059 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:15:48,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:15:48,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:15:48,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:15:48,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:15:48,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42165.98 MB 2025-02-15 00:15:48,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50600.60 MB 2025-02-15 00:15:48,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-15 00:15:48,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55446.60 MB 2025-02-15 00:15:48,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63831.02 MB 2025-02-15 00:15:48,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-15 00:15:48,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50600.60 MB 2025-02-15 00:15:48,225 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-15 00:15:48,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:15:48,227 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:15:48,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:15:48,228 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:15:48,232 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:15:48,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:15:48,233 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:15:48,234 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:16:47,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:16:47,430 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:16:47,435 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:16:47,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:16:47,439 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1683, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:16:47,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:16:47,440 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1683, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:17:13,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:17:13,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:17:13,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.98 seconds 2025-02-15 00:17:13,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:13,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45044.49 MB 2025-02-15 00:17:13,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51000.53 MB 2025-02-15 00:17:13,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5956.04 MB 2025-02-15 00:17:13,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72215.43 MB 2025-02-15 00:17:13,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58546.19 MB 2025-02-15 00:17:13,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13669.24 MB 2025-02-15 00:17:13,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59952.94 MB 2025-02-15 00:17:13,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:17:13,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:17:13,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-15 00:17:13,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:13,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51000.53 MB 2025-02-15 00:17:13,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44875.61 MB 2025-02-15 00:17:13,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6124.92 MB 2025-02-15 00:17:13,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58546.19 MB 2025-02-15 00:17:13,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 76720.11 MB 2025-02-15 00:17:13,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18173.92 MB 2025-02-15 00:17:13,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68001.82 MB 2025-02-15 00:17:15,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:17:15,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:17:15,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-15 00:17:15,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:15,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44875.61 MB 2025-02-15 00:17:15,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45406.45 MB 2025-02-15 00:17:15,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:17:15,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76720.11 MB 2025-02-15 00:17:15,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54003.76 MB 2025-02-15 00:17:15,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22716.35 MB 2025-02-15 00:17:15,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49385.00 MB 2025-02-15 00:17:15,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:17:15,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:17:15,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:17:15,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:15,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45406.45 MB 2025-02-15 00:17:15,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47295.99 MB 2025-02-15 00:17:15,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:17:15,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54003.76 MB 2025-02-15 00:17:15,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54005.86 MB 2025-02-15 00:17:15,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 00:17:15,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48713.42 MB 2025-02-15 00:17:15,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:17:15,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:17:15,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:17:15,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:15,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47295.99 MB 2025-02-15 00:17:15,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49537.84 MB 2025-02-15 00:17:15,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:17:15,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54005.86 MB 2025-02-15 00:17:15,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58252.59 MB 2025-02-15 00:17:15,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 00:17:15,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55082.12 MB 2025-02-15 00:17:15,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:17:15,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:17:15,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-15 00:17:15,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:15,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45406.45 MB 2025-02-15 00:17:15,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49537.84 MB 2025-02-15 00:17:15,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:17:15,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54003.76 MB 2025-02-15 00:17:15,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58252.59 MB 2025-02-15 00:17:15,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4248.83 MB 2025-02-15 00:17:15,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55082.12 MB 2025-02-15 00:17:16,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:17:16,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:17:16,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-15 00:17:16,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:16,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51071.38 MB 2025-02-15 00:17:16,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51838.39 MB 2025-02-15 00:17:16,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:17:16,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58252.59 MB 2025-02-15 00:17:16,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58669.92 MB 2025-02-15 00:17:16,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:17:16,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52546.17 MB 2025-02-15 00:17:16,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:17:16,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:17:16,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:17:16,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:16,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52251.28 MB 2025-02-15 00:17:16,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52479.77 MB 2025-02-15 00:17:16,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.49 MB 2025-02-15 00:17:16,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58669.92 MB 2025-02-15 00:17:16,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58669.92 MB 2025-02-15 00:17:16,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:17:16,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52697.45 MB 2025-02-15 00:17:16,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:17:16,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:17:16,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.66 seconds 2025-02-15 00:17:16,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:16,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39180.78 MB 2025-02-15 00:17:16,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52680.18 MB 2025-02-15 00:17:16,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13499.40 MB 2025-02-15 00:17:16,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72215.43 MB 2025-02-15 00:17:16,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58669.92 MB 2025-02-15 00:17:16,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13545.50 MB 2025-02-15 00:17:16,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52697.45 MB 2025-02-15 00:17:16,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:17:16,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:17:16,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:17:16,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:16,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52680.18 MB 2025-02-15 00:17:16,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44175.16 MB 2025-02-15 00:17:16,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8505.02 MB 2025-02-15 00:17:16,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58669.92 MB 2025-02-15 00:17:16,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58669.92 MB 2025-02-15 00:17:16,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:17:16,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55183.55 MB 2025-02-15 00:17:16,391 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-15 00:17:16,392 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:17:16,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:17:16,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:17:16,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:17:16,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:17:16,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44175.16 MB 2025-02-15 00:17:16,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52586.67 MB 2025-02-15 00:17:16,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8411.52 MB 2025-02-15 00:17:16,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58669.92 MB 2025-02-15 00:17:16,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62851.65 MB 2025-02-15 00:17:16,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-15 00:17:16,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52586.67 MB 2025-02-15 00:17:16,556 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-15 00:17:16,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:17:16,558 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:17:16,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:17:16,559 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:17:16,563 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:17:16,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:17:16,564 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:17:16,565 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:18:25,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:18:25,772 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:18:25,777 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:18:25,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:18:25,781 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1694, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:18:25,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:18:25,782 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1694, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:18:51,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:18:51,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:18:51,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.18 seconds 2025-02-15 00:18:51,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:51,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45121.14 MB 2025-02-15 00:18:51,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51116.89 MB 2025-02-15 00:18:51,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5995.76 MB 2025-02-15 00:18:51,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71215.09 MB 2025-02-15 00:18:51,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58560.87 MB 2025-02-15 00:18:51,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12654.22 MB 2025-02-15 00:18:51,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60029.59 MB 2025-02-15 00:18:52,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:18:52,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:18:52,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 00:18:52,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:52,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51116.89 MB 2025-02-15 00:18:52,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44932.80 MB 2025-02-15 00:18:52,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6184.10 MB 2025-02-15 00:18:52,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58560.87 MB 2025-02-15 00:18:52,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 72420.95 MB 2025-02-15 00:18:52,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13860.08 MB 2025-02-15 00:18:52,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64850.42 MB 2025-02-15 00:18:54,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:18:54,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:18:54,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-15 00:18:54,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44932.80 MB 2025-02-15 00:18:54,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45463.64 MB 2025-02-15 00:18:54,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:18:54,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72420.95 MB 2025-02-15 00:18:54,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53980.69 MB 2025-02-15 00:18:54,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18440.26 MB 2025-02-15 00:18:54,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49442.18 MB 2025-02-15 00:18:54,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:18:54,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:18:54,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:18:54,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45463.64 MB 2025-02-15 00:18:54,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47353.17 MB 2025-02-15 00:18:54,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:18:54,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53980.69 MB 2025-02-15 00:18:54,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53982.79 MB 2025-02-15 00:18:54,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 00:18:54,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48770.60 MB 2025-02-15 00:18:54,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:18:54,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:18:54,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:18:54,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47353.17 MB 2025-02-15 00:18:54,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49595.03 MB 2025-02-15 00:18:54,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:18:54,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53982.79 MB 2025-02-15 00:18:54,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58229.52 MB 2025-02-15 00:18:54,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-15 00:18:54,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55139.31 MB 2025-02-15 00:18:54,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:18:54,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:18:54,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:18:54,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45463.64 MB 2025-02-15 00:18:54,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49595.03 MB 2025-02-15 00:18:54,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:18:54,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53980.69 MB 2025-02-15 00:18:54,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58229.52 MB 2025-02-15 00:18:54,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4248.83 MB 2025-02-15 00:18:54,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55139.31 MB 2025-02-15 00:18:54,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:18:54,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:18:54,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 00:18:54,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51128.57 MB 2025-02-15 00:18:54,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51895.57 MB 2025-02-15 00:18:54,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:18:54,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58229.52 MB 2025-02-15 00:18:54,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58646.86 MB 2025-02-15 00:18:54,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:18:54,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52603.36 MB 2025-02-15 00:18:54,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:18:54,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:18:54,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:18:54,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52308.46 MB 2025-02-15 00:18:54,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52536.93 MB 2025-02-15 00:18:54,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-15 00:18:54,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58646.86 MB 2025-02-15 00:18:54,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58646.86 MB 2025-02-15 00:18:54,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:18:54,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52769.72 MB 2025-02-15 00:18:54,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:18:54,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:18:54,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.64 seconds 2025-02-15 00:18:54,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39219.11 MB 2025-02-15 00:18:54,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52737.34 MB 2025-02-15 00:18:54,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13518.23 MB 2025-02-15 00:18:54,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71215.09 MB 2025-02-15 00:18:54,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58646.86 MB 2025-02-15 00:18:54,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12568.23 MB 2025-02-15 00:18:54,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52769.72 MB 2025-02-15 00:18:54,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:18:54,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:18:54,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:18:54,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52737.34 MB 2025-02-15 00:18:54,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44213.48 MB 2025-02-15 00:18:54,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8523.86 MB 2025-02-15 00:18:54,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58646.86 MB 2025-02-15 00:18:54,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58646.86 MB 2025-02-15 00:18:54,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:18:54,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55240.71 MB 2025-02-15 00:18:54,713 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-15 00:18:54,713 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:18:54,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:18:54,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:18:54,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:18:54,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:18:54,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44213.48 MB 2025-02-15 00:18:54,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52624.30 MB 2025-02-15 00:18:54,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-15 00:18:54,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58646.86 MB 2025-02-15 00:18:54,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58646.86 MB 2025-02-15 00:18:54,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:18:54,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52624.30 MB 2025-02-15 00:18:54,880 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-15 00:18:54,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:18:54,882 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:18:54,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:18:54,883 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:18:54,887 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:18:54,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:18:54,888 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:18:54,888 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:19:59,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:19:59,323 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:19:59,328 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:19:59,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:19:59,332 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1792, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:19:59,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:19:59,333 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1792, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:20:27,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:20:27,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:20:27,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.77 seconds 2025-02-15 00:20:27,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:27,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45804.02 MB 2025-02-15 00:20:27,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52145.80 MB 2025-02-15 00:20:27,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6341.79 MB 2025-02-15 00:20:27,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67010.30 MB 2025-02-15 00:20:27,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58904.81 MB 2025-02-15 00:20:27,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8105.49 MB 2025-02-15 00:20:27,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61165.46 MB 2025-02-15 00:20:27,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:20:27,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:20:27,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 00:20:27,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:27,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52145.80 MB 2025-02-15 00:20:27,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45442.27 MB 2025-02-15 00:20:27,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6703.54 MB 2025-02-15 00:20:27,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58904.81 MB 2025-02-15 00:20:27,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 80266.40 MB 2025-02-15 00:20:27,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21361.59 MB 2025-02-15 00:20:27,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 70794.03 MB 2025-02-15 00:20:29,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:20:29,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:20:29,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-15 00:20:29,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45442.27 MB 2025-02-15 00:20:29,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45973.11 MB 2025-02-15 00:20:29,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:20:29,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 80266.40 MB 2025-02-15 00:20:29,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53978.60 MB 2025-02-15 00:20:29,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26287.80 MB 2025-02-15 00:20:29,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49951.66 MB 2025-02-15 00:20:29,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:20:29,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:20:29,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:20:29,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45973.11 MB 2025-02-15 00:20:29,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47862.64 MB 2025-02-15 00:20:29,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:20:29,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53978.60 MB 2025-02-15 00:20:29,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53980.69 MB 2025-02-15 00:20:29,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-15 00:20:29,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49280.07 MB 2025-02-15 00:20:29,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:20:29,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:20:29,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:20:29,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47862.64 MB 2025-02-15 00:20:29,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50104.50 MB 2025-02-15 00:20:29,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:20:29,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53980.69 MB 2025-02-15 00:20:29,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59171.14 MB 2025-02-15 00:20:29,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-15 00:20:29,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55648.78 MB 2025-02-15 00:20:29,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:20:29,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:20:29,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:20:29,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45973.11 MB 2025-02-15 00:20:29,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50104.50 MB 2025-02-15 00:20:29,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:20:29,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53978.60 MB 2025-02-15 00:20:29,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59171.14 MB 2025-02-15 00:20:29,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-15 00:20:29,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55648.78 MB 2025-02-15 00:20:29,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:20:29,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:20:29,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 00:20:29,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51638.04 MB 2025-02-15 00:20:29,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52405.04 MB 2025-02-15 00:20:29,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:20:29,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59171.14 MB 2025-02-15 00:20:29,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 00:20:29,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:20:29,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53112.83 MB 2025-02-15 00:20:29,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:20:29,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:20:29,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:20:29,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52817.93 MB 2025-02-15 00:20:29,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53046.13 MB 2025-02-15 00:20:29,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.20 MB 2025-02-15 00:20:29,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59588.48 MB 2025-02-15 00:20:29,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 00:20:29,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:20:29,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53288.80 MB 2025-02-15 00:20:29,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:20:29,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:20:29,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.27 seconds 2025-02-15 00:20:29,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39560.54 MB 2025-02-15 00:20:29,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53246.24 MB 2025-02-15 00:20:29,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13685.70 MB 2025-02-15 00:20:29,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67010.30 MB 2025-02-15 00:20:29,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 00:20:29,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7421.82 MB 2025-02-15 00:20:29,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53288.80 MB 2025-02-15 00:20:29,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:20:29,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:20:29,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:20:29,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53246.24 MB 2025-02-15 00:20:29,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44550.50 MB 2025-02-15 00:20:29,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8695.75 MB 2025-02-15 00:20:29,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59588.48 MB 2025-02-15 00:20:29,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 00:20:29,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:20:29,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55745.93 MB 2025-02-15 00:20:29,897 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-15 00:20:29,898 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 00:20:29,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:20:29,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:20:29,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:20:29,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:20:29,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44550.50 MB 2025-02-15 00:20:29,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52948.93 MB 2025-02-15 00:20:29,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8398.43 MB 2025-02-15 00:20:29,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59588.48 MB 2025-02-15 00:20:29,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59588.48 MB 2025-02-15 00:20:29,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:20:29,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52948.93 MB 2025-02-15 00:20:30,060 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-15 00:20:30,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:20:30,062 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:20:30,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:20:30,063 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:20:30,067 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:20:30,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:20:30,068 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:20:30,069 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 00:20:39,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:20:39,283 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:20:39,288 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:20:39,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:20:39,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1341, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:20:39,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:20:39,293 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1341, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:21:00,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:21:00,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:21:00,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.95 seconds 2025-02-15 00:21:00,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:00,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42661.38 MB 2025-02-15 00:21:00,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47407.23 MB 2025-02-15 00:21:00,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4745.85 MB 2025-02-15 00:21:00,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67939.34 MB 2025-02-15 00:21:00,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57296.29 MB 2025-02-15 00:21:00,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10643.05 MB 2025-02-15 00:21:00,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56210.07 MB 2025-02-15 00:21:00,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:21:00,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:21:00,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-15 00:21:00,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:00,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47407.23 MB 2025-02-15 00:21:00,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43097.66 MB 2025-02-15 00:21:00,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4309.57 MB 2025-02-15 00:21:00,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57296.29 MB 2025-02-15 00:21:00,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66374.86 MB 2025-02-15 00:21:00,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9078.57 MB 2025-02-15 00:21:00,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61099.64 MB 2025-02-15 00:21:02,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:21:02,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:21:02,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-15 00:21:02,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43097.66 MB 2025-02-15 00:21:02,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43628.50 MB 2025-02-15 00:21:02,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-15 00:21:02,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66374.86 MB 2025-02-15 00:21:02,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48368.71 MB 2025-02-15 00:21:02,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18006.15 MB 2025-02-15 00:21:02,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47607.05 MB 2025-02-15 00:21:02,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:21:02,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:21:02,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:21:02,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43628.50 MB 2025-02-15 00:21:02,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45518.03 MB 2025-02-15 00:21:02,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-15 00:21:02,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48368.71 MB 2025-02-15 00:21:02,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50258.25 MB 2025-02-15 00:21:02,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-15 00:21:02,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46935.46 MB 2025-02-15 00:21:02,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:21:02,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:21:02,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-15 00:21:02,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45518.03 MB 2025-02-15 00:21:02,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47759.89 MB 2025-02-15 00:21:02,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-15 00:21:02,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50258.25 MB 2025-02-15 00:21:02,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56864.28 MB 2025-02-15 00:21:02,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-15 00:21:02,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53304.17 MB 2025-02-15 00:21:02,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:21:02,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:21:02,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-15 00:21:02,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43628.50 MB 2025-02-15 00:21:02,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47759.89 MB 2025-02-15 00:21:02,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-15 00:21:02,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48368.71 MB 2025-02-15 00:21:02,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56864.28 MB 2025-02-15 00:21:02,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8495.56 MB 2025-02-15 00:21:02,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53304.17 MB 2025-02-15 00:21:02,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:21:02,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:21:02,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-15 00:21:02,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49293.43 MB 2025-02-15 00:21:02,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50060.43 MB 2025-02-15 00:21:02,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-15 00:21:02,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56864.28 MB 2025-02-15 00:21:02,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57281.61 MB 2025-02-15 00:21:02,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-15 00:21:02,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50768.22 MB 2025-02-15 00:21:02,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:21:02,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:21:02,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:21:02,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50473.32 MB 2025-02-15 00:21:02,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50700.60 MB 2025-02-15 00:21:02,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.28 MB 2025-02-15 00:21:02,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57281.61 MB 2025-02-15 00:21:02,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57281.61 MB 2025-02-15 00:21:02,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:21:02,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50914.01 MB 2025-02-15 00:21:02,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:21:02,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:21:02,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.41 seconds 2025-02-15 00:21:02,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37989.22 MB 2025-02-15 00:21:02,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50900.96 MB 2025-02-15 00:21:02,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12911.74 MB 2025-02-15 00:21:02,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67939.34 MB 2025-02-15 00:21:02,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57281.61 MB 2025-02-15 00:21:02,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10657.73 MB 2025-02-15 00:21:02,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50914.01 MB 2025-02-15 00:21:02,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:21:02,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:21:02,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-15 00:21:02,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:02,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50900.96 MB 2025-02-15 00:21:02,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42982.86 MB 2025-02-15 00:21:02,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7918.10 MB 2025-02-15 00:21:02,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57281.61 MB 2025-02-15 00:21:02,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57281.61 MB 2025-02-15 00:21:02,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:21:02,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53403.72 MB 2025-02-15 00:21:03,000 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-15 00:21:03,001 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:21:03,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:21:03,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:21:03,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:21:03,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:21:03,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42982.86 MB 2025-02-15 00:21:03,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51392.87 MB 2025-02-15 00:21:03,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.01 MB 2025-02-15 00:21:03,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57281.61 MB 2025-02-15 00:21:03,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65640.86 MB 2025-02-15 00:21:03,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-15 00:21:03,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51392.87 MB 2025-02-15 00:21:03,163 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-15 00:21:03,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:21:03,165 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:21:03,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:21:03,166 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:21:03,170 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:21:03,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:21:03,171 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:21:03,171 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-15 00:23:24,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:24,713 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:23:24,722 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:23:24,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:24,730 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:23:24,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:24,732 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:23:27,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:23:27,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:23:27,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.89 seconds 2025-02-15 00:23:27,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:27,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34599.22 MB 2025-02-15 00:23:27,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35250.38 MB 2025-02-15 00:23:27,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-15 00:23:27,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74000.11 MB 2025-02-15 00:23:27,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36779.85 MB 2025-02-15 00:23:27,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37220.25 MB 2025-02-15 00:23:27,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44071.85 MB 2025-02-15 00:23:27,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:23:27,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:23:27,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:23:27,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:27,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35250.38 MB 2025-02-15 00:23:27,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35448.04 MB 2025-02-15 00:23:27,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.66 MB 2025-02-15 00:23:27,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36779.85 MB 2025-02-15 00:23:27,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39214.65 MB 2025-02-15 00:23:27,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2434.79 MB 2025-02-15 00:23:27,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37628.62 MB 2025-02-15 00:23:28,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-15 00:23:28,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-15 00:23:28,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.80 seconds 2025-02-15 00:23:28,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35448.04 MB 2025-02-15 00:23:28,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35669.66 MB 2025-02-15 00:23:28,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-15 00:23:28,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39214.65 MB 2025-02-15 00:23:28,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37727.76 MB 2025-02-15 00:23:28,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1486.88 MB 2025-02-15 00:23:28,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39619.76 MB 2025-02-15 00:23:28,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-15 00:23:28,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-15 00:23:28,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:23:28,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35669.60 MB 2025-02-15 00:23:28,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36458.81 MB 2025-02-15 00:23:28,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 789.21 MB 2025-02-15 00:23:28,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37727.76 MB 2025-02-15 00:23:28,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38518.39 MB 2025-02-15 00:23:28,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 790.63 MB 2025-02-15 00:23:28,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37050.59 MB 2025-02-15 00:23:28,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-15 00:23:28,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-15 00:23:28,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-15 00:23:28,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36458.81 MB 2025-02-15 00:23:28,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37395.09 MB 2025-02-15 00:23:28,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.27 MB 2025-02-15 00:23:28,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38518.39 MB 2025-02-15 00:23:28,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40883.98 MB 2025-02-15 00:23:28,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2365.59 MB 2025-02-15 00:23:28,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39710.84 MB 2025-02-15 00:23:28,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-15 00:23:28,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-15 00:23:28,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-15 00:23:28,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35669.60 MB 2025-02-15 00:23:28,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37395.09 MB 2025-02-15 00:23:28,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1725.49 MB 2025-02-15 00:23:28,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37727.76 MB 2025-02-15 00:23:28,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40883.98 MB 2025-02-15 00:23:28,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3156.21 MB 2025-02-15 00:23:28,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39710.84 MB 2025-02-15 00:23:28,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-15 00:23:28,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-15 00:23:28,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-15 00:23:28,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38035.34 MB 2025-02-15 00:23:28,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38355.56 MB 2025-02-15 00:23:28,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.22 MB 2025-02-15 00:23:28,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40883.98 MB 2025-02-15 00:23:28,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41058.04 MB 2025-02-15 00:23:28,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-15 00:23:28,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38658.66 MB 2025-02-15 00:23:28,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-15 00:23:28,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-15 00:23:28,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-15 00:23:28,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38527.95 MB 2025-02-15 00:23:28,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38751.13 MB 2025-02-15 00:23:28,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.18 MB 2025-02-15 00:23:28,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41058.04 MB 2025-02-15 00:23:28,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41058.04 MB 2025-02-15 00:23:28,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:23:28,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38769.79 MB 2025-02-15 00:23:28,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:23:28,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:23:28,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.91 seconds 2025-02-15 00:23:28,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33958.14 MB 2025-02-15 00:23:28,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38951.71 MB 2025-02-15 00:23:28,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4993.57 MB 2025-02-15 00:23:28,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74000.11 MB 2025-02-15 00:23:28,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41058.04 MB 2025-02-15 00:23:28,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32942.06 MB 2025-02-15 00:23:28,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38951.71 MB 2025-02-15 00:23:28,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-15 00:23:28,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-15 00:23:28,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-15 00:23:28,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38951.71 MB 2025-02-15 00:23:28,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37855.13 MB 2025-02-15 00:23:28,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1096.59 MB 2025-02-15 00:23:28,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41058.04 MB 2025-02-15 00:23:28,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41058.04 MB 2025-02-15 00:23:28,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-15 00:23:28,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39553.04 MB 2025-02-15 00:23:28,927 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-15 00:23:28,927 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-15 00:23:28,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-15 00:23:28,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-15 00:23:28,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-15 00:23:28,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:23:28,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37855.13 MB 2025-02-15 00:23:28,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46273.28 MB 2025-02-15 00:23:28,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-15 00:23:28,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41058.04 MB 2025-02-15 00:23:28,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51520.73 MB 2025-02-15 00:23:28,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-15 00:23:28,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46273.28 MB 2025-02-15 00:23:29,089 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-15 00:23:29,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:29,091 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-15 00:23:29,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:29,092 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-15 00:23:29,096 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-15 00:23:29,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:29,097 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-15 00:23:29,097 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-15 00:23:57,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:57,543 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-15 00:23:57,548 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-15 00:23:57,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:57,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3345, 3, 384, 384]), torch.float32, cuda:0] 2025-02-15 00:23:57,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-15 00:23:57,553 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3345, 3, 378, 378]), torch.float32, cuda:0] 2025-02-15 00:24:49,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-15 00:24:49,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-15 00:24:49,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.77 seconds 2025-02-15 00:24:49,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:24:49,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 56626.26 MB 2025-02-15 00:24:49,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 68464.69 MB 2025-02-15 00:24:49,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11838.42 MB 2025-02-15 00:24:49,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 87386.23 MB 2025-02-15 00:24:49,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70986.50 MB 2025-02-15 00:24:49,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16399.73 MB 2025-02-15 00:24:49,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 80303.11 MB 2025-02-15 00:24:49,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-15 00:24:49,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-15 00:24:49,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-15 00:24:49,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:24:49,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 68464.69 MB 2025-02-15 00:24:49,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 91240.94 MB 2025-02-15 00:24:49,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 22776.25 MB 2025-02-15 00:24:49,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70986.50 MB 2025-02-15 00:24:49,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 92742.35 MB 2025-02-15 00:24:49,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21755.85 MB 2025-02-15 00:24:49,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 97168.44 MB 2025-02-15 00:24:49,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-15 00:24:49,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-15 00:24:49,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 52.07 seconds 2025-02-15 00:24:49,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-15 00:24:49,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44971.67 MB 2025-02-15 00:24:49,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 91240.94 MB 2025-02-15 00:24:49,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 46269.27 MB 2025-02-15 00:24:49,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75730.26 MB 2025-02-15 00:24:49,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 92742.35 MB 2025-02-15 00:24:49,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17012.10 MB 2025-02-15 00:24:49,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 97168.44 MB